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ABSTRACT 



The Austin Collaborative for Mathematics Education (ACME) is 
a districtwide initiative to improve mathematics education in all elementary 
and middle school classrooms in the Austin Independent School District, Texas 
(AISD) . The initiative, funded by the National Science Foundation and the 
school district, provides long-term, high-quality professional development to 
build the instructional capacity of more than 2,000 AISD mathematics 
teachers. This evaluation of ACME effectiveness was based on student 
mathematics test results from the Texas Assessment of Academic Skills (TAAS) 
and Iowa Tests of Basic Skills (ITBS) ; mathematics results; observations of 
48 mathematics lessons and 7 professional development sessions; principal 
(n=88) and teacher (n=250) questionnaires, interviews with 10 teachers, AISD 
staff and administrators; and other AISD documents. As assessed by the ITBS, 
student basic mathematics knowledge has remained steady since the 
implementation of the ACME project. The percentage of students passing the 
TAAS rose from the 1998-1999 passing rates for most groups. Larger gains were 
seen for African American, Hispanic, and economically disadvantaged students 
even though those students continued to lag behind White students. ACME staff 
provided teachers with high quality, long-term professional development that 
was particularly effective in helping teachers who were not experienced with 
standards-based instruction learn how to use the designated curriculum 
resources. However, ACME professional development alone has not generally 
helped teachers who achieve a moderate level of competence become strong 
implementers of standards-based instruction. Recommendations are made for 
program improvement. Appendixes contain a discussion of gains and losses in 
TAAS mathematics scores, a chart of changes in professional development and 
implementation, and the evaluation instruments. (Contains 22 figures, 1 
table, and 13 references.) (SLD) 
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Executive Summary 
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The Austin Collaborative for Mathematics Education (ACME) is a districtwide initiative to improve 
mathematics education in all elementary and middle school classrooms in the Austin Independent School 
District (AISD). This initiative, funded by the National Science Foundation (NSF) and the district, provides 
long-term, high quality professional development to build the instructional capacity of over 2000 AISD 
mathematics teachers. ACME professional development supports teachers as they implement the district’s 
curriculum resources of Investigations in Number , Data , and Space and Connected Mathematics (CMP), which 
are aligned with the state standards for mathematics education in the Texas Essential Knowledge and Skills 
(TEKS) and the national standards set by the National Council of Teachers of Mathematics (NCTM). These 
standards focus on broadening the topics taught at all grade levels, developing children’s mathematical thinking, 
and deepening children’s conceptual understanding through concrete experiences. The standards contrast with 
traditional mathematics education which is characterized by rote memorization and computation practice. 

ACME professional development is designed to help teachers deepen their knowledge of mathematics 
content and standards-based pedagogy as well as to grow as a community of learners. Every elementary and 
middle school mathematics teacher, including general education, special education, and bilingual teachers, is 
expected to participate in two years of summer institutes and follow-up days during the academic year. To 
promote districtwide change, the ACME project focuses on the development of professional school cultures, 
administrative and teacher leadership, and community and parental involvement. 

Major Findings 

The evaluation of ACME effectiveness was based on student TAAS and ITBS mathematics results; 
observations of mathematics lessons and professional development sessions; principal and teacher 
questionnaires; interviews with teachers, ACME staff, and district administrators; and other AISD documents. 

• The percentage of students passing the 1999-2000 TAAS mathematics rose from the 1998-1999 passing 
rates for most groups. African American, Hispanic, and economically disadvantaged students made larger 
gains than did White students, although the scores remained lower than the scores of White students. 

• Strong implementation of standards-based mathematics instruction was related to the highest student TAAS 
mathematics passing rates, to the highest mean TLI scores (scaled scores to permit comparison across years 
and across grades), and to the highest passing rates for each of the 13 TAAS mathematics objectives. 
Standards-based mathematics instruction prepared students to pass the four problem-solving objectives 
particularly well. Students’ problem-solving skills will be essential to passing future versions of TAAS. 

• As assessed by the ITBS, student basic mathematics knowledge has remained steady since the 
implementation of the ACME project. 

• ACME staff provided teachers high quality, long-term professional development. ACME professional 
development has been effective in helping teachers who are not experienced with standards-based 
instruction learn how to use the designated curriculum resources. However, ACME professional 
development alone has not generally helped teachers who achieve a moderate level of competence become 
strong implementers of standards-based instruction. The improvement of teachers’ pedagogical skills and 
content knowledge was somewhat limited. 
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• Effective campus support for teacher implementation of standards-based mathematics (e.g., coaching that 
focuses on mathematics content, mentoring, and collaborative planning) is still in its infancy in AISD. 

• Since the inception of the ACME project, changes in district, campus, and project leadership have yielded 
mixed messages, unclear vision, and wavering support for the implementation of standards-based 
mathematics at AISD. The AISD dual textbook adoption also sent mixed messages about district goals for 
mathematics education, although the focus on the state standards TEKS has redressed some confusion. 

• Persistent concerns about students’ passing the state assessment, TAAS, has continued to distract some 
teachers from implementing standards-based mathematics, despite strategies to address these concerns. 

Recommendations 

1. Enlist district administrators to communicate a clear message about the district's vision for mathematics 
education because mixed messages have fostered piecemeal implementation of standards-based instruction 
across the district. Broadcast the message on the AISD cable channel to reach teachers, campus 
administrators, parents, and community members. In area principal meetings, include 10 minute updates on 
the mathematics program (e.g., attendance at ACME professional development, TEKS and TAAS 
mathematics objectives, and the association between standards-based instruction and student achievement). 

2. Make explicit the connections between ACME and other district initiatives, especially IFL, because the 
approaches to teaching and learning are compatible. IFL is an opportunity to strengthen the instructional 
leadership of district and campus administrators, which is a weak link in AISD’s implementation of 
standards-based mathematics. Making the connections explicit should foster a shared vision for AISD’s 
direction in curriculum and instruction and bolster necessary administrative support. If AISD is not able to 
bolster administrative support for standards-based mathematics instruction, it should look at other 
mathematics programs. 

3. Hire and train campus instructional specialists who are skilled in standards-based mathematics instruction 
through AFL funding. Establish collaborative relationships between these specialists and ACME 
facilitators to provide a network of strong support for implementation on campuses. Concentrate this 
campus support on cognitive coaching and content-focused collaboration. By developing effective forms of 
campus support, AISD will help more teachers become strong implementers of standards-based 
mathematics instruction, which is linked to high levels of student achievement on TAAS mathematics 
(especially problem-solving skills that will be key to passing future versions of TAAS). 

4. Provide new ACME staff with professional development to maintain the quality of ACME professional 
development for teachers. To ease the transition in ACME staff, develop cognitive coaching among team 
members and routinely examine teacher evaluations of ACME professional development to devise strategies 
to improve facilitators’ skills. 
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Project Overview 

In August of 1997, the Austin Independent School District (AISD) launched the Austin 
Collaborative for Mathematics Education (ACME) initiative to improve mathematics education in all 
elementary and middle school classrooms using standards-based curriculum resources and 
instruction. The National Science Foundation (NSF) and AISD funded the initiative, which is a 
collaborative with the Charles A. Dana Center and the University of Texas at Austin. In the 1998-99 
school year, the ACME project served over 2000 AISD educators who teach about 55,000 students at 
71 elementary and 17 middle schools in a district of approximately 77,000 students (46% Hispanic, 
17% African American, 35% Anglo and 2% other; AISD Office of Student Services, Sept. 2000). 
The ACME project is unique because it serves every elementary and middle school mathematics 
teacher in a large urban district with long-term professional development. 

The ACME project builds the instructional capacity of all mathematics teachers by providing 
a minimum of 120 hours of professional development through summer institutes and follow-up 
sessions. Some teachers also participate in campus level support, such as lesson modeling and 
collaborative planning. The intent of ACME professional development is to build teachers’ capacity 
to deliver effective mathematics instruction to all students, to ensure consistent implementation of 
quality mathematics curriculum resources across the district, and to provide ongoing support for 
teachers and administrators as they implement standards-based curriculum and instruction. 
Specifically, district staff design ACME professional development to help teachers grow as a 
community of learners and to deepen their knowledge of mathematics content, pedagogy, and 
classroom management for inquiry-based mathematics instruction. 

ACME provides every elementary and middle school mathematics teacher, including general 
education, special education, bilingual, and English as a Second Language (ESL) teachers, the 
opportunity to participate in a series of professional development activities lasting two years. 
Participants begin their training with a summer institute lasting two weeks and continue with four to 
five follow-up days during the academic year. The second year involves a three-day summer institute 
and three to four follow-up days. Teachers are paid a stipend to attend the summer institutes and 
follow-up sessions outside school hours, and substitutes are provided to release teachers during the 
academic year. 

ACME professional development began working with teachers at the transition between 
elementary and middle school so that students would have consistent mathematics instruction from 
one year to the next. In the summer of 1997, fifth and sixth grade teachers began ACME 
professional development, followed by fourth and seventh grade teachers in the summer of 1998, 
second, third, and eighth grade teachers in the summer of 1999. Most kindergarten and first grade 
teachers began ACME professional development in the summer of 2000. Some kindergarten and 
first grade teachers, who were not yet targeted for implementation, chose to attend two days of 
professional development during the 1999-2000 school year because the district adopted the 
standards-based texts in the spring of 1999. 

To accommodate the needs of AISD teachers and administrators, ACME staff adjusted the 
original design of ACME by adding professional development sessions on Saturdays and evenings, 
designing sessions for special education teachers, and adding overviews for late hires. To address 
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teacher turn-over (more than 500 new hires yearly), ACME staff continued to offer summer institutes 
and follow-up for teachers new to the district or who had not yet participated. 

At most schools in the district, AISD implemented ACME professional development by 
grade levels. Yet, at eight pilot elementary schools, teachers of all grade levels participated in 
ACME professional development simultaneously. Three pilot middle schools participated in the 
NSF-funded State Systemic Initiative (SSI) beginning with sixth grade mathematics teachers in the 
summer of 1996. Pilot schools received modified summer institutes: fewer days of summer institutes 
and follow-up sessions, in exchange for campus support such as modeling lessons and conversations 
about curriculum and instruction. In the 1999-2000 school year, ACME staff continued to work with 
one pilot school that requested ongoing support. 

The district supplies rigorous curriculum resources to support the mathematics instructional 
capacity of teachers as part of the ACME initiative. The resources are based on standards set by the 
National Council of Teachers of Mathematics (NCTM, 1989, 1991, 1995), by the state in the Texas 
Essential Knowledge and Skills (TEKS), and by AISD’s Mathematics Department in the local 
curriculum document. In the spring of 1999, the district adopted the curriculum resources of 
Investigations in Number ; Data , and Space for elementary grades and Connected Mathematics 
{CMP) for middle grades, and purchased these materials to support teachers’ implementation of 
standards-based instruction. AISD also adopted the resources of Math in My World (English 
version)/ Mathematicas in Mi Mundo (Spanish version) for elementary grades and Mathematics: 
Applications and Connections , Courses 1-3 (English version)/ Mathematicas: Aplicaciones y 
Coneciones , Cursos 1-3 (Spanish version) to supplement TEKS areas not addressed in Investigations 
and CMP. This adoption ensures that all of AISD’s mathematics education resources and efforts are 
consistent with local, state, and national standards. 

The curriculum resources of Investigations and CMP are well suited for the ACME initiative 
compared to traditional textbooks because they support the following teaching practices: 

• Promoting children’s mathematical thinking, reasoning, and problem-solving skills; 

• Developing children’s deep understanding of mathematical concepts through concrete 
experiences, real-world problems, and communication; and 

• Supporting a vertically and horizontally coordinated curriculum that addresses the needs 
of all students, including those who are served by the special education, bilingual, and 
gifted and talented programs (Russell, 1998). 

These practices emphasize children’s mathematical literacy by promoting the understanding 
of mathematics concepts and approach instruction through problem-solving and communication of 
ideas. These practices contrast with traditional practices that emphasize mathematical algorithms, 
rote memorization, and computation mastery (Cohen & Ball, 1990). 

To promote districtwide change in mathematics education, the ACME project bolsters 
leadership and the development of school cultures in which communities continually improve 
mathematics teaching and learning. ACME staff provide institutes for campus administrators to 
build knowledge of standards-based mathematics curriculum resources and instruction and to help 
campus leaders develop strategies for supporting teachers in implementation. ACME staff also work 
with other organizational structures in AISD that promote teacher leadership (e.g., curriculum 
specialists) to support the continuous improvement of mathematics education on campuses. In 
addition, the ACME project has customized professional development for teacher leaders so that they 
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may facilitate sessions and support their peers at the campus level in a variety of ways, including 
peer coaching, demonstration teaching, and information sharing. To gamer parent participation in 
the mathematics curriculum, the project staff provides schools with deliverables (e.g., pamphlets and 
videos in English and Spanish) as well as assistance with organizing parent education and 
involvement (e.g., parent math nights). Additionally, the project staff enlists program support from 
AISD’s administrative leaders. 
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Impact on Student Mathematics Achievement 

The impact of the ACME project on student mathematics achievement is central to 
evaluating its effectiveness. While ACME activities focus on intensive professional development for 
teachers, improving student learning is a major goal of ACME. 

Student Results and Teacher Implementation of Standards-Based Mathematics 

To examine the direct effects of curriculum and instruction on student mathematics 
achievement, associations between the quality of teacher implementation of standards-based 
mathematics and student scores on the Texas Assessment of Academic Skills (TAAS) and the Iowa 
Test of Basic Skills (ITBS) were analyzed. 

Classroom Observations and the Quality of Implementation 

In the spring of 2000, evaluators observed the mathematics lessons of 48 teachers, including 
teachers in 10 bilingual and three special education classrooms. Forty of the 48 teachers were first 
randomly selected and observed in the spring of 1998 or in the spring of 1999; eight additional 
teachers were randomly drawn in the spring of 2000. AISD evaluators and Dana Center staff were 
trained and certified to reliably rate the quality of implementation of standards-based mathematics 
education on an 8-point ordinal scale using the HRI Classroom Observation Protocol (HRI, 1999a; 
see Appendix C). Most of the classroom observations (over 90%) were in elementary classrooms 
because the sampling frame of all AISD mathematics teachers includes more elementary school 
teachers than middle school teachers. 

The quality of implementation of standards-based mathematics was simplified to three 
categories: Weak implementation, moderate implementation, and strong implementation. 
Observers discussed the concepts underlying the 8-point scale of the HRI protocol to determine the 
subcategories. 

Weak implementation refers to lessons that show little evidence of standards-based 
instruction. Students passively received information from the teacher or were involved in activities 
that lacked purpose and were unlikely to enhance mathematical thinking. Moderate implementation 
occurred when observers found evidence of the beginning stages of standards-based teaching 
strategies that engaged students in problem-solving, but the quality of the lesson was limited. The 
lesson may have lacked teaching strategies that pushed students to deep understandings, or may have 
muddled conceptual knowledge with inaccurate or superficial exploration of mathematics content. 
Strong implementation refers to lessons that observers coded as effective and engaging standards- 
based instruction that helped most students successfully solve mathematical problems and developed 
conceptual understanding. 

Student TAAS Mathematics Results 

The TAAS is a state-mandated, criterion-referenced test. TAAS measures student mastery of 
the state standards TEKS in mathematics at grades 3 through 8 and at exit level. (Reading, writing, 
science, and social studies are also tested, but not all subjects are administered at all grade levels.) 

The TAAS results are presented as the percentage of students passing, the percentage of 
students passing each of 13 mathematics objectives, and the mean (or average) Texas Learning Index 
(TLI). The TAAS mathematics objectives are divided into three domains: Objectives 1 through 5 



ERIC 



12 



99.14 



Austin Collaborative for Mathematics Education. Annual Report. 1999-2000 



ERIC 



are designed to assess Concepts; Objectives 6 through 9 assess Operations; and Objectives 10 
through 13 assess Problem-Solving. The TLI is a scaled score that permits comparison across years 
and across grades. A TLI score of 70 is considered passing, and indicates that a student meets 
minimum expectations and is in line to meet the exit level standard if current progress continues. 
Student TAAS Mathematics Results and the Quality of Teacher Implementation 

Student TAAS mathematics results were combined for the 30 classrooms out of the 48 
observed in the spring of 2000, including bilingual and special education classes. The sample was 
limited to 30 classrooms because only grades 3 through 8 were tested on TAAS. Eleven of the 
lessons were rated as weak implementation, eight were rated as moderate implementation, and eleven 
were rated as strong implementation. The percentage of students receiving free and reduced-price 
lunch varied by teacher implementation: 60% in lessons rated as weak implementation; 55% in 
lessons rated as moderate implementation; and 40% in lessons rated as strong implementation. 

Figure 1 presents the percentages of students passing the test and each objective, and Figure 
2 presents the mean TLI in the observed classrooms by the quality of teacher implementation of 
standards-based mathematics. 

The associations between student TAAS mathematics data and the quality of teacher 
implementation of standards-based mathematics suggest the following: 

• Student mathematics achievement was higher in classrooms with strong implementation in 
all analyses than was student achievement in classrooms with weak and moderate 
implementation. 

• Moderate implementation was associated with higher student achievement than was weak 
implementation in the Problem-Solving Domain, Objectives 10 through 13, and in Algebra 
and Measurement. 

• Weak implementation was associated with higher student achievement than was moderate 
implementation in the Operations Domain, Objectives 6 through 9, and in Geometry. 
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Figure 1. Percentage of Students Passing TAAS Mathematics by Quality of Teacher 

Implementation in Spring of 2000 1 




Passing Objective 1, Objective 2, Objective 3, Objective 4, Objective 5, 
Number Concepts Algebra Geometry Measurement Probability & 

Statistics 




Objective 6, Objective 7, Objective 8, Objective 9, 
Addition Subtraction Multiplication Division 




Objective 10, Objective 11, Objective 12, Objective 13, 

Estimation Solution Strategies Representation Reasonableness 



El Weak □ Moderate H Strong 



Note: For Objectives 1-8, the number of students in classrooms rated as weak implementation = 189; the 
number of students in classrooms rated as moderate implementation = 141; and the number of students in 
classrooms rated as strong implementation = 239. For Objectives 9-13, the numbers of students are smaller: 
TEA decided to collapse some TAAS Objectives for grades 3 and 4 due to limited exposure to some topics at 
those grade levels. 



1 Chi-square tests were statistically significant (p < .01) indicating that the number of students passing TAAS 
mathematics and passing each of the 13 objectives varied significantly by the quality of teacher implementation. 
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Figure 2. Mean TLI for Students in TAAS Mathematics by Quality of Teacher 
Implementation in Spring of 2000 




El Weak □ Moderate H Strong 



Note: The number of students in classrooms rated as weak implementation = 189; the number of students in 
classrooms rated as moderate implementation = 141; and the number of students in classrooms rated as strong 
implementation = 239. 

Student ITBS Mathematics Results 

The ITBS is a norm-referenced test of general educational achievement that is administered 
to all AISD students at grades 3, 5, and 8 only. The ITBS assesses a wide range of skills including 
higher-order thinking skills, interpretation, classification, comparison, analysis, and inference. AISD 
students were administered two of three ITBS mathematics subtests: Concepts and Estimation, 
Problem-Solving and Data Interpretation, but not Computation. The ITBS results are presented as 
percentile ranks of the average standard score. A percentile rank of 50 indicates that 50% of all 
students who took the test nationally scored below that score. 

Student ITBS Mathematics Results and the Quality of Teacher Implementation 

Student ITBS mathematics results were combined for 15 classrooms of the 48 observed in 
the Spring of 2000, including bilingual and special education classes. The sample was limited to 15 
classrooms because only students in grades 3, 5, and 8, were tested on ITBS. 

The ITBS results are presented as the percentile rank of the average standard score. Figure 3 
presents the percentile rank for the students who were tested and enrolled in the observed classrooms 
in the Spring of 2000. 

The association between student ITBS mathematics data and the quality of teacher 
implementation of standards-based mathematics suggests the following: 

• Student mathematics achievement was associated with the quality of implementation. 

• Students in classrooms with strong implementation scored higher than students in classrooms 
with moderate or weak implementation. 
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Figure 3. Percentile Rank of Students Tested in ITBS Mathematics by Quality of Teacher 

Implementation in Spring of 2000 




El Weak □ Moderate H Strong 

Note: The number of students in classrooms rated as weak implementation = 48; the number of students in 
classrooms rated as moderate implementation = 37; and the number of students in classrooms rated as strong 
implementation = 67. 

District Mathematics Results 
District TAAS Mathematics Results 

To examine the global impact of the ACME project on AISD student mathematics 
achievement, district Texas Assessment of Academic Skills (TAAS) mathematics results are 
presented. The results for all AISD students tested were taken from the Texas Education Agency 
(TEA) Summary Reports for this evaluation. The data include scores of students who took the 
English version of the test, not the Spanish version; students in year-round schools; and students 
enrolled in special education classes, except in the 1997-1998 school year. 

TAAS mathematics results are presented by grade and by disaggregated accountability 
student groups for the 1997-98, 1998-99, and 1999-2000 school years. The results for students in 
grades 3 through 8 are included because these grade levels are targeted by ACME. (Kindergarten 
through grade 2, although targeted by ACME, however are not tested with TAAS.) The results are 
presented by disaggregated groups; the groups are African American, Hispanic, White, and 
economically disadvantaged students. TEA differentiates student performance by these groups to 
hold districts and campuses accountable for the achievement of all students on all campuses. 

The TAAS results are presented in two ways: (1) the percentage of students passing (i.e., a 
TLI score of 70 or above) and (2) the mean TLI (see explanation, “Student TAAS Mathematics 
Results,” p. 4). Figures 4 through 15 present the percentages passing TAAS mathematics and the 
mean TLI for grades 3 through 8 and disaggregated groups in 1997-98, 1998-99, and 1999-2000. The 
number of years of implementation of standards-based mathematics varied by grade level. By the 
1999-2000 school year, teachers in grades 5 and 6 had been implementing for three years, teachers in 
grades 4 and 7 had been implementing for two years, and teachers in grades 3 and 8 had been 
implementing for one year. 

It is important to note that the influence of standards-based curriculum and instruction on the 
district TAAS and ITBS mathematics results is confounded by observed lessons that were 
supplemented with materials that were neither standards-based nor recommended by the district’s 
Mathematics Department. 
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AISD student performance on the 1999-2000 TAAS mathematics in comparison with the 
1998-99 results suggest the following observations: 

• The percentage of students passing TAAS mathematics increased for the majority of student 
groups, except for students in grade 3, even though students served by special education are 
included in the results after 1997-98. 

• The mean TLI in mathematics increased for nearly every group across all grade levels. 

• African American, Hispanic, and economically disadvantaged students made larger gains in 
mean TLI and in passing rates than White students (see Appendix A for gains and losses by 
disaggregated groups), although the results of African American, Hispanic, and economically 
disadvantaged students continued to be lower than the scores of White students. 

• Middle school students made larger gains in mean TLI and in passing rates than did 
elementary students. 

• Cohort analysis suggests that achievement gains made in grades 7 and 8 may be attributable 
to three years of standards-based mathematics instruction and ACME. 
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Figure 4. Percentage of Students in Grade 3 Passing TAAS Mathematics in 1997-98, 1998-99, 

and 1999-2000 




Disadvantaged 



H 1997-98 □ 1998-99 ^1999-2000 



Note: The data for the 1 997-98 school year did not include students in special education. The number of 
students each year were: 1997-98, n = 4488; 1998-99, n = 4995; and 1999-2000, n = 4867. 



Figure 5. Mean TLI for Students in Grade 3 in TAAS Mathematics in 1997-98, 1998-99, and 

1999-2000 




U 1997-98 □ 1998-99 @1999-2000 



Note: The data for the 1997-98 school year did not include students in special education. The number of 
students each year were: 1997-98, n = 4488; 1998-99, n = 4995; and 1999-2000 n = 4867. 
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Figure 6. Percentage of Students in Grade 4 Passing TAAS Mathematics in 1997-98, 1998-99, 

and 1999-2000 




All Students African American Hispanic White Economically 

Disadvantaged 



M 1997-98 □ 1998-99 W 1999-2000 



Note: The data for the 1997-98 school year did not include students in special education. The number of 
students each year were: 1997-98, n = 4540; 1998-99, n = 4936; and 1999-2000, n = 5058. 



Figure 7. Mean TLI for Students in Grade 4 in TAAS Mathematics, 1997-98, 1998-99, and 

1999-2000 




All Students African American Hispanic White Economically 

Disadvantaged 



m 1997-98 □ 1998-99 ^1999-2000 



Note: The data for the 1997-98 school year did not include students in special education. The number of 
students each year were: 1997-98, n = 4540; 1998-99, n = 4936; and 1999-2000, n = 5058. 
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Figure 8. Percentage of Students in Grade 5 Passing TAAS Mathematics in 1997-98, 1998-99, 

and 1999-2000 




All Students African American Hispanic White Economically 

Disadvantaged 
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Note: The data for the 1997-98 school year did not include students in special education. The number of 
students each year were: 1997-98, n = 4416; 1998-99, n = 5102; and 1999-2000, n = 4797 . 



Figure 9. Mean TLI for Students in Grade 5 in TAAS Mathematics in 1997-98, 1998-99, and 

1999-2000 




All Students African American Hispanic White Economically 

Disadvantaged 



M 1997-98 □ 1998-99 ^1999-2000 



Note: The data for the 1997-98 school year did not include students in special education. The number of 
students each year were: 1997-98, n = 4416; 1998-99, n = 5102; and 1999-2000, n = 4797. 
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Figure 10. Percentage of Students in Grade 6 Passing TAAS Mathematics in 1997-98, 1998-99, 

and 1999-2000 




All Students African American Hispanic White Economically 

Disadvantaged 
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Note: The data for the 1997-98 school year did not include students in special education. The number of 
students each year were: 1997-98, n = 4202; 1998-99, n = 4738; and 1999-2000, n = 4894. 



Figure 11. Mean TLI for Students in Grade 6 in TAAS Mathematics in 1997-98, 1998-99, and 

1999-2000 




All Students African American Hispanic White Economically 

Disadvantaged 



M 1997-98 □ 1998-99 ^1999-2000 



Note: The data for the 1997-98 school year did not include students in special education. The number of 
students each year were: 1997-98, n = 4202; 1998-99, n = 4738; and 1999-2000, n = 4894. 
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Figure 12. Percentage of Students in Grade 7 Passing TAAS Mathematics in 1997-98, 1998-99, 

and 1999-2000 




All Students African American Hispanic White Economically 

Disadvantaged 
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Note: The data for the 1997-98 school year did not include students in special education. The number of 
students each year were: 1997-98, n = 4286; 1998-99, n = 4623; 1999-2000, n = 4621. 



Figure 13. Mean TLI for Students in Grade 7 in TAAS Mathematics in 1997-98, 1998-99, and 

1999-2000 
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Note: The data for the 1997-98 school year did not include students in special education. The number of 
students each year were: 1997-98, n = 4286; 1998-99, n = 4623; 1999-2000, n = 4621. 
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Figure 14. Percentage of Students in Grade 8 Passing TAAS Mathematics in 1997-98, 1998-99, 

and 1999-2000 




All Students African American Hispanic White Economically 

Disadvantaged 



M 1997-98 □ 1998-99 ^1999-2000 



Note : The data for the 1997-98 school year did not include students in special education. The number of 
students each year were: 1997-98, n = 4156; 1998-99, n = 4654; 1999-2000, n = 4466. 



Figure 15. Mean TLI for Students in Grade 8 in TAAS Mathematics in 1997-98, 1998-99, and 

1999-2000 




American Disadvantaged 
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Note: The data for the 1997-98 school year did not include students in special education. The number of 
students each year were: 1997-98, n = 4156; 1998-99, n = 4654; 1999-2000, n = 4466. 
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District ITBS Mathematics Results 

An argument against standards-based instruction is that students’ mathematics achievement 
will decline because the emphasis on problem-solving may not provide opportunities to learn 
mathematics facts. TAAS measures students’ knowledge of the state standards TEKS, which are 
consistent with the mathematical content and process standards of the AISD curriculum resources. 
To examine further the impact of the ACME project on students’ mathematics achievement, district 
Iowa Tests of Basic Skills (ITBS) results are presented by grades tested for the three years of 
implementation of ACME. The ITBS is nationally-normed, assesses broader range of knowledge 
than TAAS, and allows for comparison with student scores nationwide. 

The ITBS mathematics results for all AISD students enrolled at grades 3, 5, and 8 are 
analyzed in this report. Figure 16 presents the percentile rank (for explanation, see “Student ITBS 
Mathematics Results,” p. 7) for the average performance of all AISD students in grades 3, 5, and 8 
who took the test in the 1997-98, 1998-99, and 1999-2000 school years. Teachers in grade 5 were 
targeted for implementation of standards-based instruction for all of the three years presented, while 
teachers in grades 3 and 8 were targeted for implementation only in the 1999-2000 school year. 

AISD student performance on the ITBS suggests the following observations: 

• Mathematics achievement has remained steady since the inception of the ACME project. 

• Grade level comparisons show that grade 3 has performed slightly below the national 
average, while grades 5 and 8 have performed slightly above the national average. 

• Implementation of the ACME project appears to have neither helped nor hindered student 
achievement on the ITBS. 

Figure 16. Percentile Rank of Students Tested in ITBS Mathematics, 1997-98, 1998-99, 1999- 

2000 




Grade 3 Grade 5 Grade 8 



0 1997-98 01998-99 0 1999-00 



Note: In 1997-98, Grade 3 (n=5363), Grade 5 (n=5716), and Grade 8 (n=5267); in 1998-99 Grade 3 (n=5634), 
Grade 5 (n=5859), and Grade 8 (n=4998); and in 1999-2000 Grade 3 (n=5634), Grade 5 (n=5540), and Grade 8 
(n=5138). 
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Quality of ACME Professional Development 

The key activity of the ACME project to improve mathematics instruction districtwide is 
intensive professional development for teachers. This section provides a description of ACME 
professional development and an analysis of the impact the project has had on mathematics teachers 
and standards-based instruction in AISD classrooms in the 1999-2000 school year. 

Sources 

Professional Development Observations 

The information for this analysis came from several sources. The lead evaluator observed 7 
ACME professional development sessions throughout the 1999-2000 school year, and formally rated 
five of these sessions on an 8-point scale using the HRI Professional Development Observation 
Protocol (HRI, 1999b; see Appendix C). Five were formally rated to meet NSF requirements, and 
additional sessions were informally observed to supplement the information. 

Teacher Interviews 

Ten randomly selected mathematics teachers, most of whom (8 of 10) had participated in 60 
or more hours of ACME professional development, completed phone interviews. The interviews 
included questions about teachers’ thoughts and feelings about ACME professional development, 
changes in practice, and school and district policies that facilitate or hinder reforms in mathematics 
education (see Appendix C). 

Teacher Questionnaires 

A random sample of 300 AISD elementary and middle school mathematics teachers were 
sent questionnaires, and 250 teachers of the 266 eligible returned valid questionnaires (return rate, 
88%). One-third (34%) had taught school for 5 years or less, one-third (31%) had taught for 6 to 15 
years, and one-third (34%) had taught for 16 years or more. The Local Systemic Change (LSC) 
Teacher Questionnaires surveyed teachers’ beliefs about mathematics instruction, preparation, 
classroom practice, mathematics content knowledge, perceptions of district support, and experiences 
in ACME professional development (see Appendix C). 

Principal Questionnaires 

The 88 AISD middle schools and elementary principals completed LSC Prinicpal 
Questionnaires about standards-based mathematics and ACME professional development (see 
Appendix C). 

Additional Sources 

Additional sources of information included interviews with district and ACME project staff, 
observations of district and project meetings, district and state mathematics curriculum documents, 
professional development materials, brochures, letters, and newsletters. 

ACME Professional Development Facilitators 
Composition of ACME Professional Development Team 

In the third year of the project, the organization of ACME professional development 
facilitators was similar to that of the previous year. A core team of six ACME facilitators supported 
by the NSF grant provided the bulk of the ACME professional development and support to teachers. 
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Two district administrators and one district mathematics specialist supported the initiative by 
working with teachers and principals on campuses, by providing ACME professional development in 
the summer, and by observing the day to day realities of implementing the curriculum resources. 

As in previous years, CMP facilitators from Michigan were hired to provide middle school 
summer institutes. Follow-up sessions during the academic year for middle school teachers were 
provided by one of the six ACME facilitators and a liaison with the Dana Center at the University of 
Texas. 

A consultant with Marilyn Bums’ Math Solutions provided additional professional 
development to a cadre of teachers and ACME staff as in the previous year. ACME staff invited 
teachers who appeared to be highly motivated to implement standards-based curriculum resources 
and expressed deep understanding of standards-based pedagogy to participate. The cadre was 
expanded from 40 teachers in the previous year to 80 teachers. In addition to elementary and middle 
school teachers, project staff added high school teachers. The cadre sessions focused on 
mathematical content knowledge (i.e., algebraic thinking, geometry, and vertical links from 
elementary content to calculus), spheres of influence for leading standards-based instruction, 
discourse in the classroom, and content-focused coaching. Several teachers who participated in the 
cadre helped provide professional development for summer institutes by modeling lessons and 
sharing their classroom experiences implementing the resources. 

Changes in ACME Professional Development Team 

At the beginning of the 1999-2000 school year, the ACME project lost a charismatic leader, 
an original designer and cheerleader for the grant, and has been struggling to recapture its original 
vigor. By the end of the summer of 2000, four ACME professional development facilitators had left 
the project for other positions because they were no longer wanted to work on the ACME team. Five 
new professional development facilitators were hired. Most of the new facilitators were participants 
in the ACME teacher cadre and teachers fresh out of the classroom. One new facilitator had 
extensive experience providing professional development and campus support with a New York 
Local Systemic Change (LSC) initiative. At the end of the school year, only one original member of 
the core ACME team remained, and many new members were still getting acclimated to the work. 

In the summer of 2000, the district divided its mathematics curriculum team into secondary 
and elementary teams. The interim ACME project director, who had been a district mathematics 
specialist for three years, led the secondary team and a new leader was hired from outside the district 
to lead the elementary team and to supervise the ACME project. 

Format of ACME Professional Development 
Design of Support for Teachers 

ACME professional development for teachers consisted of weeklong summer institutes and 
follow-up days during the academic year. Follow-up days included sessions during school, after 
school, and on Saturdays. In the 1999-2000 school year, professional development was held at the 
district’s Professional Development Academy (PDA) and at an additional site to meet the needs of 
teachers who live and work in north as well as in south Austin. 

ACME professional development facilitators continued to integrate mathematics content 
knowledge, pedagogy, and the use of curriculum resources into the summer institutes and follow-up 
days as before. The ACME project did not hold separate sessions to focus on mathematics content 
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knowledge. Although a professional development session on cognitive coaching was offered for the 
first time in the fall of 1999, it was canceled because only two teachers in the district had registered. 
Campus Support 

Ongoing support to teachers implementing the curriculum resources generally took the form 
of follow-up days held at PDA. Few teachers received support on campuses. Several ACME 
professional development facilitators visited a handful of campuses, but the visits were short-term. 

In the previous year, campus support was limited to teachers at several pilot schools (i.e., 
eight campuses that implemented standards-based curriculum and instruction in all grade levels 
simultaneously). In the 1999-2000 school year, campus support was the charge of two ACME 
facilitators. Each of the two facilitators selected five campuses to visit weekly for half a day. They 
met with second and third grade teachers who were in their first year of implementation of standards- 
based mathematics, about four teachers per campus. The two facilitators also visited 20 additional 
campuses on when requested. 

To design a model of campus support, the two ACME facilitators collaborated with a 
colleague from a New York LSC and with ACME staff who had provided campus support with pilot 
schools the year before with the ACME evaluator. These facilitators selected several schools with 
low student passing rates on TAAS mathematics and schools whose teachers were highly engaged in 
the 1999 ACME summer institutes and showed motivation to implement standards-based curriculum 
and instruction. Included in the plan were strategies for establishing rapport with campus staff and 
guidelines for principals about the purpose of visits. These facilitators ended campus support in the 
fall because the ACME project needed staff to provide professional development sessions. 
Additionally, the ACME campus support facilitators perceived a lack of interest from teachers and 
administrators and found that visits lacked meaning (which ACME staff coined as the “parade 
wave”). They believed that developing trust was key to establishing a professional dialogue on 
campuses, which takes time, perhaps a year. One facilitator said, “To go into classrooms you need to 
build trust before you can begin talking. People who need help either don’t know they need it or 
don’t want it. It’s like going into somebody’s home.” 

Another ACME facilitator explored a model of campus support with one pilot school in 
which teachers and administrators wanted to continue professional development after completing the 
two years of summer institutes and follow-up days. The “Collaborative Assessment” model focused 
on improving instruction by examining student work and organizing content-focused conversations 
among colleagues. This approach appeared more effective than the one described in the previous 
paragraph because it focused discussion and reflection on student learning. It also reportedly 
refocused conversations in the teachers’ lounge on teaching and learning mathematics. This focus on 
student learning also is a major goal of the Institute for. Learning (IFL) a district initiative to improve 
leadership. 

The differences in effectiveness of these two approaches to campus support centered on three 
elements: the school climate, the facilitator’s skill level, and the model. When the school climate 
consisted of teachers and administrators who were knowledgeable about standards-based 
mathematics and motivated to improve instruction as in the case of the pilot school, the professional 
dialogue reached more campus staff than when the school climate was characterized by a lack of 
interest in changing instruction. The facilitator at the pilot school had honed her skills in guiding 
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teachers’ conversations about student learning and professional development for several years, 
whereas the other facilitators were less skilled in supporting teachers. The model of campus support 
provided structured discussions of teaching and learning, whereas the model at other campuses 
focused on brief, superficial discussions about how implementation of standaids-based mathematics 
instruction was progressing. Thus, in the 1999-2000 school year, effective campus support 
apparently occurred on one AISD campus. 

In sum, developing campus cultures that provide ongoing support for teachers in their 
classroom, an original goal of the NSF grant, is still in its infancy. Campus support was not 
structured, rarely focused on mathematics content and pedagogy, and reached few teachers. 
Teachers’ standing requests for observation and feedback were not systematically addressed. The 
ACME project offered to help teachers develop cognitive coaching relationships, but teachers did not 
appear ready for the opportunity. On the basis of a recent study of implementation of standards- 
based curriculum and instruction on AISD campuses (Batchelder & Christian, 1999), the synergism 
necessary for meaningful professional development to manifest on campuses is not yet common in 
the district. 

Preparation of ACME Professional Development Facilitators 
Orientation to ACME Professional Development 

All professional development facilitators were former classroom teachers who were campus 
leaders in standards-based curriculum and instruction, and many had provided professional 
development for district, state, and national organizations. To orient new facilitators to the project in 
previous years, new members built on the expertise of established ACME facilitators by observing 
professional development sessions before facilitating their own sessions. However, in the 1999-2000 
school year, orientation to the ACME project was skipped in part because most new facilitators 
learned about the project through the teacher cadre. New facilitators were assigned sessions, 
provided notebooks with professional development pieces, and received little guidance on the ACME 
approach to developing learning communities and to the needs of teachers. (In the fall of 2000, new 
professional development facilitators are again taking time to observe experienced ACME facilitators 
and become oriented to the project.) ACME facilitators continued to participate in national 
conferences for professional development such as the Technical Educational Research Center 
(TERC) leadership conference, “Administrators as Leaders, Parents as Partners,” and the conference, 
“Diversity, Equity, and Standards, An Urban Agenda in Mathematics Education,” sponsored by NSF, 
NCTM, and New York University. Additional professional development for ACME facilitators 
included “Effective Strategies for Engaging Teachers in Staff Development” and “Quality of 
Implementation of Standards-Based Instruction” provided by district staff. 

Communication of ACME Professional Development Facilitators 

In the third year of ACME, changes in central office and project leadership brought changes 
in how ACME facilitators worked together and communicated. Communication from district leaders 
and among ACME leaders was segregated from other ACME staff. Communication among ACME 
facilitators changed from a focus on improving ACME professional development to concerns about 
personal needs. For example, although lunch breaks previously were times for reflection and 
debriefing among ACME facilitators, in the summer of 2000 conversations focused on changes in 
district and uncertainty about the direction of ACME project organization. Thus, changes in the 
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district and the project had an impact on time spent reflecting and improving the effectiveness of 
ACME professional development. 

Shared Vision of ACME Professional Development Facilitators 

Many of the ACME professional development facilitators continued to hold a shared vision 
of the goals of the project: The vision, as one ACME facilitator reported, focused on improving 
mathematics instruction with “professional development at the center... for really getting teachers 
excited about teaching mathematics, empowering them to work as a team, and really learn how to 
implement the curriculum,... to get teachers to take over leadership roles, and to see the bigger 
picture.” Yet, during the third year of the ACME project, talk that questioned the value of standards- 
based curriculum and instruction emerged among AISD mathematics specialists. In addition, rather 
than directly supporting standards-based mathematics instruction, district administrators emphasized 
teaching the curriculum embedded in the state standards TEKS, students’ knowledge of which the 
TAAS assesses. Consideration of resources, other than Investigations and CMP, that prepared 
students for TAAS also surfaced. Although most ACME facilitators valued standards-based 
instruction to improve mathematics education, a shift in emphasis destabilized the vision. 

CULTURE OF ACME PROFESSIONAL DEVELOPMENT 
Development of a Learning Community 

In the third year of the ACME project, the culture of ACME professional development 
focused on developing a learning community. To lay the groundwork for the learning community, 
ACME facilitators established norms on the basis of national standards for staff development and 
teacher feedback. These norms, posted and discussed at ACME professional development, included: 
(a) honor our time; (b) take responsibility for your learning and the learning of others; (c) focus on 
the purpose; and (d) keep student learning at the forefront. The goal was to make respect for 
colleagues explicit and to emphasize adult and student learning. 

An introductory ACME professional development session for kindergarten and first grade 
teachers exemplified how the norms worked. The facilitator launched the session by starting on time 
stating, “I’m going to honor your time.” Participants spent several minutes discussing the question, 
“Why do we come to professional development?” The facilitator commented that much of the 
discussion focused on the challenges of implementing the standards-based curriculum resources (e.g., 
reading the teacher books, organizing materials), but not on student thinking. This comment guided 
participants to turn to a discussion of student learning. 

The facilitator, then asked for feedback on the discussion, which encouraged participants to 
reflect on the process of professional development and to be open about their reactions. One woman 
thought it was “helpful to realize that other people are going through the same things I am.” After 
sharing stories about personal experiences learning mathematics, one woman stated “If a lot of the 
same things come up, you could just list it. It would take less time.” Another woman responded, 
‘This discussion reinforces my belief in a balance of manipulatives and drill.” Thus, the facilitator 
guided teachers in sharing opinions that were supportive as well as oppositional to the professional 
development activities and to reform in mathematics instruction. 

In ACME professional development, teachers and facilitators shared their struggles and 
insights about implementing standards-based mathematics in their classrooms. In a kindergarten and 
first grade summer institute, teachers and the facilitators participated in a book study of Growing 



me 



23 



29 



99.14 



Austin Collaborative for Mathematics Education , Annual Report, 1999-2000 



Mathematical Ideas in Kindergarten (Schulman-Dacey & Eston, 1999). One teacher expressed the 
challenges of changing teaching practices and said, “I have problems going from rote [instruction] to 
exploring deeper.” The facilitator set the tone for reflection by conceding that questioning is “what’s 
hard about Investigations .” He figured out questioning strategies were “the reason my kids weren’t 
making the ‘Aha.’” He then tied that discovery to the participants’ success in a problem-solving 
activity from that week of ACME professional development. He said, “This is the first time I felt the 
groups understood the ‘Swimming Pool Problem,”’ to which he attributed his development of 
effective questioning strategies. 

Although most professional development facilitators focused on developing a learning 
community, the quality of facilitation varied across sessions, as was seen in previous years. In one 
observed ACME professional development session, for example, the facilitator directed the 
discussion in ways that seemed unresponsive to teachers’ needs, which seemed to alienate some 
participants. Yet, despite or perhaps because of this apparent unresponsiveness, several teachers in 
the session added focus and leadership to the discussion by sharing their experiences implementing 
standards-based curriculum and their beliefs about reforming mathematics instruction. While 
variability in the quality of facilitation may hamper teachers’ experiences in ACME professional 
development, some participants’ motivation to implement standards-based mathematics may endure 
and influence others. 

Levels of Engagement in ACME Professional Development Activities 

In the third year of the ACME project, more teachers were observed to be actively engaged 
in professional development activities than before. However, in some observed sessions, 25% of the 
participants were not actively engaged (e.g., were discussing campus politics, grading papers) as in 
previous years. Some participants arrived in late (up to 30 minutes), as before, which was not 
consistent with commitment to the “honor our time” norm for professional development. 

While some ACME facilitators employed effective strategies for engaging participants, 
others appeared to disregard the issue. Effective strategies included: (a) validating and giving voice 
to a variety of opinions by summarizing what participants said during break out sessions; (b) 
changing seating arrangements daily to mix participants from across the district; (c) using name 
sticks to draw out participants and to encourage every participant to be responsible for learning; and 
(d) talking to participants during breaks, including unfamiliar faces and quiet ones. Ineffective 
strategies included not talking to teachers that did not seem engaged and asking teachers to hold their 
comments without returning to the points later in a session. It appeared that making the norms 
explicit in dialogue with participants throughout sessions was more effective than simply posting the 
norms and presenting them once in a session. 

Relevance of ACME Professional Development 

Some lack of engagement in ACME professional development activities may be due to some 
teachers’ not finding relevance in ACME professional development. In interviews, teachers 
expressed positive and negative beliefs about ACME professional development. As in the past, some 
teachers were impatient with the structure of sessions. One teacher reported that ACME professional 
development “could be faster; you do activities, and a lot of talking between is a waste of time; I’d 
like not to go.” Other teachers wanted more time spent learning games of Investigations . One 
teacher said, “I would have spent more time on games and not put much theory into it, [I’d spend] 
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more time on individual book activities.” Other teachers expressed positive experiences in ACME 
professional development. One teacher said, “It was really helpful to plan as a team.... Working 
with [an ACME facilitator] was more helpful than playing the games.” Another teacher stated, 
“They’re doing a good job, and they’re good at answering people’s questions.... I learn much more 
with CMP than kill kids with drill.” 

Although some teachers did not find activities of ACME professional development relevant, 
attitudes in general have remained lukewarm. On the basis of the LSC Teacher Questionnaire, over 
half of the teachers surveyed (57%) rated the quality of ACME professional development as “good,” 
“very good,” or “excellent,” while less than one third of respondents (29%) rated it “fair,” and a 
small proportion (14%) rated it “poor” or “very poor.” The overall quality of rating of ACME 
professional development declined slightly in the Spring of 2000 from the Spring of 1999. 
Deepening Teachers’ Understanding of Mathematics Content 

The ACME approach to deepening teachers’ understanding of mathematics content 
continued in the project’s third year as in previous years.. Mathematics content was infused 
throughout ACME professional development. The approach included the following components: 

• ACME facilitators presented engaging problems to provide opportunities for participants to 
explore mathematics deeply and to reflect on their experiences as adult learners and compare 
their experiences to those of students. 

• ACME professional development activities asked teachers to examine children’s 
mathematical thinking and problem solving strategies (e.g., videos presenting student 
strategies for solving multiplication and division problems and the derivation of what 
students need to know to solve these problems). 

• While working with the curriculum resources, ACME professional development addressed a 
variety of content areas such as number sense, computation strategies, measurement, 
algebraic thinking, and geometry (e.g., how children learn to count from the Investigations 
Teacher Notes); probability and statistics were not covered. 

Thus, placing student mathematical thinking at the forefront of professional development 
discussions was a focus of ACME professional development in the third year. Although this 
approach appeared to make mathematics content accessible to a number of teachers, for some 
teachers, gaining understanding was hit or miss. Not all content areas were explored thoroughly, nor 
was mathematics content differentiated for the needs of various teachers. 

The informal assessment of how well teachers were learning mathematics content continued 
as before through informal conversations and observations during ACME professional development. 
On the basis of responses to the LSC Teacher Questionnaire, increases in how prepared teachers felt 
to teach mathematics content that had occurred in the second year of the ACME project had 
stabilized by the third year. 

Familiarizing Teachers with Curriculum Resources and Pedagogy 
Curriculum Resources 

The approach of ACME professional development to helping teachers become familiar with 
standards-based curriculum resources and pedagogy continued as in the previous year. The approach 
to familiarizing teachers with standards-based curriculum resources included: 

• To begin, a scavenger hunt helped teachers discover parts of the curriculum resources. 
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• ACME professional development often asked participants to engage in activities with 
manipulatives, to play the games in the resources, and to explore the mathematics underlying 
the activities. In follow-up during the school year, professional development activities 
focused on books that teachers were scheduled to use in the coming months. 

• Teachers shared classroom experiences with the resources in group and panel discussions, 
including information about how to organize materials and classroom management. 

• Classroom teachers from the teacher cadre modeled lessons from Investigations and CMP 
and shared classroom experiences in summer institutes. 

• To address the needs of diverse learners teachers discussed extensions and adaptations to 
activities, and the ACME project developed charts with extensions for gifted and talented, 
special education, and bilingual/ESL students. 

Increasing Teachers 9 Standards-Based Pedagogical Knowledge 

The approach to increasing teachers’ knowledge of standards-based pedagogy included: 

• ACME facilitators modeled inquiry-based pedagogy, pointed out the questions they asked to 
push participants’ thinking to new levels, and asked teachers to discuss the strategies used to 
facilitate exploration of mathematics content and student thinking. 

• Summer institutes included a book study of Beyond Arithmetic (1995) in which teachers 
reflected on inquiry-based pedagogy, student learning, and mathematics curriculum. 

• Participants examined Bloom’s taxonomy of learning and related it to the mathematics 
TEKS. 

• ACME facilitators presented videos of AISD teachers implementing standards-based 
pedagogy and held discussions on teaching strategies and student dialogue. 

• Second grade teachers who administered the Performance Assessment in Language Arts and 
Mathematics (PALM) and who were targeted for implementation of ACME curriculum 
resources scored their students’ work with rubrics and discussed how describing the work 
could inform instruction. (Although kindergarten and first grade teachers also administered 
PALM, they were not targeted for implementation in the 1999-2000 school year.) 

• Teachers also received an extensive set of handouts with questions to promote deep 
exploration of mathematics with students. 

Observations of ACME professional development revealed variability in the depth of 
discussions and in putting into practice these approaches. While some ACME facilitators appeared 
to effectively engage participants and motivate deep exploration, other facilitators were not 
stimulating or attentive to best practices for staff development. 

On the basis of the LSC Teacher Questionnaire, the teachers surveyed continued to endorse 
standards-based teaching strategies as in previous years. Yet, their level of endorsement increased in 
the previous year and stabilized in ACME’s third year. Although the teachers surveyed continued to 
report that their pedagogical knowledge was higher than mathematics content knowledge, 
pedagogical knowledge had increased in the previous year and stabilized in the third year of the 
ACME project. 

Professional Development Tailored to Special Education Teachers 

To help special education teachers become familiar with standards-based curriculum 
resources and pedagogy, ACME professional development was expanded to include sessions tailored 
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to their needs. Special education teachers attended professional development on number sense in the 
fall and on operations in the spring, with primary and secondary teachers attending separately. A key 
feature of the session was a special education teacher who presented case studies recounting how she 
adapted one lesson to the unique learning styles of three children. The special education teachers 
who attended reported appreciation of the rare opportunity to get together and to talk about work. 

Impact of ACME Professional Development on 
Classroom Instruction 

Classroom Observations in the Spring of 2000 

Evidence of the impact of ACME professional development on instruction was derived from 
classroom observations (for a sample description, see “Classroom Observations and the Quality of 
Implementation,” p. 13). These observations provided a small, representative sample of mathematics 
instruction in the district. Although a large number of the observations (69%) included the 
curriculum resources of Investigations in Number, Data, and Space and Connected Mathematics 
(CMP) that were selected for the ACME initiative, observers remarked that a few teachers may have 
chosen to use these materials only because an ACME evaluator was observing the lesson. 

Many of the observed lessons included key elements of standards-based instruction such as 
problem-solving, communication, and using manipulatives for concrete representation, but a 
proportion of the observations involved rote activities such as drilling mathematics facts with flash 
cards. The observed lessons covered a variety of topics, including numeration and number theory, 
computation, patterns and relationships, and/or geometry. A majority of the lessons (67%) involved 
students as an entire classroom and/or individuals; thirty-eight percent involved students in small 
group activities. (Some lessons included more than one organizational structure.) Centers were used 
infrequently (19% of observations). The teachers’ stated purpose for most of the observed lessons 
(60%) was to develop or review children’s conceptual understanding, and the teachers intended 
students to learn mathematics facts in some lessons (23%). A majority of the observed lessons 
centered student activities on problem-solving (88%) and/or the use of manipulatives (54%). 
Classroom discussions occurred in many observations (42%), and in some lessons (25%) students 
answered textbook or worksheet questions. Computers, calculators, and audio-visual resources were 
used infrequently (21 % of observations). 

Definition of Rating Scale 

The quality of implementation of standards-based instruction was rated using the Classroom 
Observation Protocol (HRI, 1999b), an 8-point global scale. Previous analyses simplified these 
ratings to three categories: weak, moderate, and strong implementation. 2 On the 8-point scale, level 1 
refers to instruction that shows little evidence of student engagement with mathematical ideas. Level 
1 has two subcategories. Level 1A involves passive learning in which raters observed the students 
receiving knowledge from the teacher or text. Level IB refers to activity for activity’s sake in which 
hands-on lessons lacked purpose or content. Level 2 describes instruction that may have included 



^eak implementation includes levels 1A, IB, and 2 of the HRI Classroom Observation Protocol; moderate 
includes levels 3 low and 3 solid; and strong includes levels 3 high, 4, and 5. 
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elements of standards-based strategies but observers coded the lesson as having substantial problems 
in design, implementation or content and was limited in the likelihood to enhance children’s 
mathematical understanding. At Level 3 observers coded instruction at the beginning stages of 
standards-based teaching strategies by engaging children in mathematical concepts and problem- 
solving but may not have reached some children. Level 3 is broken down into low, solid, and high. 
Level 4 reflects standards-based instruction that was effective and engaging and appeared to help 
most students solve mathematical problems successfully. Level 5 describes exemplary instruction 
that engaged all of the students most of the time in mathematical problem-solving, communication, 
and conceptual understanding and represented the art more than the craft of teaching. 

Quality of Implementation in the Springs 1999 and 2000 

The observation ratings of the quality of implementation of standards-based mathematics 
instruction in the Springs of 1999 and 2000 were similar 3 , although the 1999-2000 school year 
brought some decline (see Figure 17). In the Spring of 2000, more lessons were rated at level 2 and 
fewer lessons were rated at level 3 low than were in the Spring of 1999. In addition, no mathematics 
lesson in the Spring of 2000 was rated at level 5. These results suggest a slight shift in the district 
away from high quality standards-based instruction. 

Figure 17. Frequencies of Observation Ratings of the Quality of Teacher Implementation for 

the Springs of 1999 and 2000 




Observation Ratings 

□ 1999 H 2000 

Source: Classroom Observations 

It is important to interpret these results cautiously. Differences in longitudinal observation 
ratings may be due to the differences in raters from one year to the next. Two AISD evaluation staff 
rated classroom observations in the Spring of 1999, and were replaced in 2000 by raters who had 
strong mathematics content backgrounds and who may have rated lessons more stringently than the 



3 Longitudinal observation ratings were correlated, r(40) = .57,/? < .001. 
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observers in 1999 4 . In addition, Horizon Research, Inc. (HRI), subcontracted by NSF to design and 
direct the national evaluation of LSC initiatives, provided intensive training viewing and rating 
classroom videos for one and a half days. HRI certified raters as reliable if their ratings of a set of 
classroom videos fell within one level of the official NSF rating. Thus, differences across years 
could also be due to the inter-rater reliability criterion. Moreover, measuring a teacher’s instruction 
on the basis of one observation per year is not reliable. An educational researcher postulated that 
frequent observation, about six ratings in one year, might provide reliable data of a teacher’s 
instructional competence (Ball, 1999). 

ACME Participation and Quality of Implementation in the Spring 2000 

The time teachers spent in ACME professional development by the Spring of 2000 appeared 
to influence the quality of implementation of standards-based mathematics instruction (see Figure 
18). The teachers observed in 56% of the 48 lessons had participated in 12 or more days of ACME 
professional development, and most of the ratings demonstrated moderate and strong levels of 
implementation of standards-based mathematics instruction (level 3 low and above). Nineteen 
percent of the teachers observed had participated in 4 to 11 days of ACME professional 
development, and the ratings centered around moderate levels of implementation of standards-based 
instruction (level 3 low). Twenty-five percent of the teachers observed had participated in 3 or fewer 
days of ACME professional development, and most of the ratings reflected weak levels of 
implementation (level 2 and below). 

Figure 18. Percentage of Observation Ratings of the Quality of Teacher Implementation by 
ACME Professional Development Days in the Spring of 2000 




Observation Ratings 



El 3 or fewer days D4-11 days B 12 or more days 



Source: Classroom Observations 



4 One 1999 observer who did not observe in the Spring of 2000 tended to rate lessons 2.5 levels above the other 
1999 observers, ANOVA, F(5, 44) = 2.09, p = .09. 
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Participation in ACME professional development appears to not be helping a number of 
teachers become competent at standards-based instruction, however. Twenty-three percent of the 
teachers whose lessons were rated as weak implementation (level 2 and below) had participated in a 
great deal of ACME professional development (4 or more days). These results are cause for concern. 
This finding may be due to ineffective professional development as well as teachers’ unwillingness 
to change their practice. On the other hand, other systemic factors may influence these results such 
as lack of administrative support on campuses for implementation, little time during the school day 
for teacher collaboration focused on mathematics content knowledge and student learning, and 
AISD’ s lack of clear vision about mathematics education. 

A few of the teachers observed (8%) had spent little time in ACME professional 
development but presented moderate or strong levels of implementation of standards-based 
instruction (level 3 low and above). As noted in a previous ACME evaluation (Batchelder & 
Christian, 1999), teachers who are “experts” in standards-based teaching practice are an untapped 
resource in AISD. These teachers could provide support such as mentoring or peer coaching on 
campuses. 

Longitudinal Classroom Observations 

Change in the Quality of Implementation of Standards-Based Instruction 

The mathematics lessons of 40 teachers were observed longitudinally, once in either 1998 or 
1999 and once in 2000. The pie chart (Figure 19) illustrates the percentage of the 40 rated lessons 
that “advanced,” “regressed,” and did not change (“no change”) in quality of teacher implementation 
of standards-based mathematics (weak, moderate and strong implementation; see Appendix B for the 
changes in observation ratings). A majority of the mathematics lessons observed (60%) did not 
change in the quality of implementation of standards-based instruction, 25% of the observed lessons 
regressed, and only 15% of the observed lessons advanced. 

Figure 19. Proportion of Observation Ratings that Advanced, Regressed, or Did Not Change in 

Quality of Teacher Implementation 




Source: Classroom Observations 

In general, the mathematics lessons of teachers whose ratings advanced by the Spring of 
2000 were not implementing standards-based teaching strategies when they were first observed. The 
advanced group was rated significantly lower at the first observation on average than were the 
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regressed group or no change group 5 . The average first rating for the advanced group was level 2. 
At level 2, instruction focuses on practicing computation and does not appear to help children deepen 
their conceptual understanding of mathematics. The average first rating was level 3 solid for lessons 
that did not change and regressed. At level 3, instruction includes many components of effective 
standards-based instruction that help children develop conceptual understanding and solve complex 
mathematical problems. 

It is important to note that the small proportion of teachers whose lessons advanced may have 
been affected by the number of teachers not continuing in the longitudinal study. Thirty-seven 
percent of the 63 teachers who were observed in either the Spring of 1998 or the Spring of 1999 did 
not participate in a second observation for various reasons (e.g., personal leave, not teaching 
mathematics, hired for other positions), and several teachers refused to continue. Additionally, the 
district has a teacher turnover rate between 15% and 20% per year, including retirees, recently 
certified teachers, and others. The teachers who did not continue to participate in the study in the 
Spring of 2000 tended to have a first observation rating that averaged one level below the rating of 
the teachers who participated longitudinally 6 . Because the average first observation rating of 
teachers whose lessons advanced was lower than the ratings of teachers whose lessons regressed or 
did not change, it is likely that if more teachers had continued the study, the size of the advanced 
group might be larger. 

ACME Professional Development Participation and Change in the Quality of Implementation 

The amount of participation in ACME professional development should relate to changes in 
the quality of the implementation of standards-based curriculum and instruction. However, the 
results were complex (see Appendix B, Table 1). Changes in observation ratings were not directly 
related to the number of ACME professional development hours attended for all of the teachers who 
participated in the longitudinal study. 

Advanced ratings. Most of the teachers whose ratings advanced (5 of 6 observations) had 
participated in 4 or more days of ACME professional development in the last year. Participation 
appeared to help some teachers who lacked knowledge and skills in standards-based instruction begin 
to develop those teaching strategies. 

No change ratings. Among the group whose ratings did not change, 46% (11 of 24 
observations) had participated in little ACME professional development (i.e., 3 or fewer days) in the 
past year. Over half of the teachers whose ratings did not change (13 of 24 observations) had 
participated in a considerable amount of ACME professional development (i.e., 4 or more days) in 
the past year. Thus, for a number of teachers, ongoing participation in ACME professional 
development did not render major improvements in standards-based teaching practices. 

Regressed ratings. Most of the teachers whose ratings regressed (7 of 10 observations) had 
participated in a considerable amount of ACME professional development (i.e., 4 to 1 1 days) in the 
past year. Thus, despite participating in ACME professional development during the 1999-2000 
school year, some teachers did not maintain or advance to higher levels of competence in standards- 
based teaching strategies. Observers noted that some regression was due to teachers’ decisions to 



5 One-way ANOVA, F (2, 37) = 3.75, p < .05. 

6 One-way ANOVA, F (1 , 61) = 3.43, p = .07; the mean levels of first observations tended to be 3 low for 
teachers who left the study and 3 solid for teachers who participated longitudinally. 
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steer away from standards-based curriculum resources (e.g., by integrating mathematics and art or by 
drilling students with flash cards to control a class in which many students had disruptive behavior). 

Caveats. The results of this longitudinal analysis should be considered cautiously. First, the 
observations reflect ratings of one day in an academic year, while many factors can influence the 
quality of instruction (e.g., mood, familiarity with the lesson, external events). Second, as noted 
above, the observers in 2000 may have rated lessons more stringently than the observers in 1999 and 
thus influenced the size of the regressed group. Additionally, observers noted that many 
observations took place after TAAS when instruction appeared to “shut down,” and the quality of 
instruction was compromised. Although some teachers may have the capacity to implement 
standards-based instruction, which is linked to student mathematics achievement, they appear to 
abandon the curriculum after testing. Consequently, AISD students may loose three to four weeks of 
quality instruction and learning. 

Generalizations from Observers 

The following generalizations of the observers inform these results: 

• Although the district has adopted the curriculum resources of Investigations in Number, 
Data, and Space and Connected Mathematics (CMP) and the supplemental texts of Math in 
My World and Mathematics: Applications and Connections, Courses 1-3, teachers were 
observed routinely supplementing lessons with materials that were not standards-based (e.g., 
Excel worksheets and Arithmetic Done Daily, A.D.D.) to drill students for TAAS. 

• Teachers did not seem to learn what makes lessons engaging from ACME professional 
development; they reduced lessons to the procedures and cut out rich activities in which 
students establish mathematical understandings. 

• Teachers have not become skilled in teaching strategies that raise the quality of instruction 
(e.g., asking questions that challenge student thinking and wrapping up lessons with key 
concepts of lessons that reinforce student learning.) 

• Teachers have not developed a complex understanding of mathematics content knowledge. 

• The difference between lessons rated at accomplished levels of standards-based instruction 
and lessons rated as lacking standards-based instruction were teacher expectations and value 
for what students would learn from the lesson. 

These generalizations support the conclusion that ACME professional development may help 
teachers who are not experienced with standards-based instruction learn how to use the high quality 
curriculum resources and develop some competence in the teaching strategies. Yet, teachers who 
develop a level of competence do not develop their skills further and become highly effective at 
standards-based instructional strategies. The ACME project has not yet helped a majority of AISD 
teachers gain the mathematics content knowledge and the pedagogical skills necessary to become 
highly effective at standards-based instruction. 

Ongoing Support to Teachers Implementing Standards-Based Mathematics 
Materials for Campuses 

Before the third year of ACME, the district had purchased curriculum resources for all grade 
levels implementing standards-based instruction (second through eighth grades) and kits for every 
two teachers implementing. In response to teacher feedback, the district supplied every teacher with 
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a kit. Additionally, the district provided packets of most student sheets for teachers implementing in 
the 1999-2000 school year to reduce teachers’ photocopying load. 

In the 1999-2000 school year, distribution of materials to teachers on campuses did not run 
smoothly. Materials for kindergarten and first grade teachers were not available from the publishers 
by the first day of classes in August. Although these grade levels were not yet targeted to implement 
the ACME-designated resources, the district adoption of Investigations required distribution. The 
student sheets were also copied and distributed to campuses for every classroom. The sheets were 
delivered a few weeks after school began. Additionally, keeping track of campus inventories with 
packing slips as well as with staff turnover continued to be problematic as in previous years. 
Follow-up Support 

Ongoing support to teachers implementing the curriculum resources generally took the form 
of follow-up days. As stated previously, on campus support was rare. As in previous years, some 
teachers found benefits in the ACME follow-up professional development during the academic year 
whereas others did not receive what they felt they needed. For example, one teacher valued working 
with the curriculum resources during follow-up professional development. She stated, “The follow- 
up training really shows you how you need to be teaching the materials; the facilitators point out 
difficulties and suggest different ways to approach the activities.... If you pick up a book without 
training, it’s very difficult.” Other teachers questioned the plan of ACME follow-up. One teacher 
stated, “In the follow-ups we didn’t get into every book; it was rushed. I think the TAAS activities 
were not relevant.” It appeared that teachers appreciated support using the materials during the 
academic year, but some disagree about how the time should be spent. While exploring the TAAS, 
TEKS, and links to standards-based resources allayed the concerns of some teachers, other did not 
see the relevance of these activities. 

Implementation of ACME Professional Development 

In the third year of the ACME project, implementation of ACME professional development 
has continued as planned. With kindergarten and first grade beginning the two year professional 
development series, all targeted grade levels, kindergarten through eighth grade, have participated on 
schedule. 

Changes in the design were instigated the previous year to provide the ACME professional 
development annually for new hires and teachers who change grade levels. Although the changes 
addressed the ongoing need for ACME professional development, many new teachers were hired just 
before school started and missed the foundation provided in ACME summer institutes. A one day 
overview provided after the first day of classes was not sufficient preparation for teachers new to 
standards-based instruction. While some struggled with implementing standards-based curriculum 
resources, others did not attempt implementation. 

Teacher attendance at ACME professional development continued for first time participants 
at rates similar to previous years, although in the summer of 2000 many teachers did not return for a 
second summer institute. For elementary, many kindergarten and first grade teachers (over 80%) 
participated in the first week of the ACME summer institute and a smaller number returned for the 
second week (70%). Similarly, approximately 85% of new second grade teachers, 70% of new third 
grade teachers, and 80% new fourth grade teachers attended the first week of their first ACME 
summer institute. Attendance dropped off in the second week for new fourth grade teachers (45% 
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returned). A large number of second and third grade teachers also did not return for their second 
ACME summer institute; only 30% of second and third grade teachers returned to complete ACME 
professional development. For middle school, while most new teachers (almost 100%) attended the 
first ACME summer institute, few middle school teachers (less than 33%) returned for the second 
summer institute. 

Support for ACME Reforms 

Changes in Leadership 

Change in district leadership has impacted the level of support for the ACME vision of 
mathematics education. The district has had a different superintendent every year since the ACME 
project began. Deputy and area superintendents as well as ACME project leadership have changed. 
Key voices that originally rallied support for changes in mathematics education are no longer AISD 
leaders. Although in the past support for changes in mathematics education advocated by ACME 
was incomplete, recent changes in leadership resulted in a set back. New district leaders need to 
become knowledgeable of the design and implementation of the ACME project as well as its 
advantages and disadvantages for teaching and learning. 

Change in district leadership has blurred the messages about the direction of mathematics 
education in the district and has yielded uncertainty on campuses. Support from campus 
administrators for the ACME vision of change in mathematics education continued to be variable 
across the district. While some campus administrators expect teachers to implement standards-based 
mathematics curriculum and instruction and structure time for teachers to collaborate and improve, 
other campus administrators do not endorse standards-based instruction and direct teachers toward 
other curriculum resources (Batchelder & Christian, 1999). Campus administrators who support 
ACME reforms organize teacher leaders to mentor other teachers as they develop standards-based 
instructional strategies, provide half-days for grade levels to collaborate on mathematics content. 
Campus administrators who do not support ACME reforms encourage teachers to use a battery of 
curriculum materials that are not standards-based, do not learn about standards-based curriculum and 
instruction, or do not communicate expectations that teachers will implement it. 

Data from the LSC Principal Questionnaires indicate that support for standards-based 
mathematics instruction has declined from high endorsement in the Spring of 1998 to moderate 
endorsement in the Spring of 2000. In the third year of ACME, fewer principals strongly agreed that 
they were knowledgeable of national standards in mathematics and well-prepared to support teachers 
implementing standards-based instruction than had in the first year of the program. The difficulty 
establishing support for standards-based instruction may be due in part to high principal turn-over 
rates in the district. Some elementary and middle school principals (41%) reported that they were 
new to the job, holding the position of principal for 3 years or less; two-thirds (66%) had been 
principal at that particular school for 3 years or less; half (52%) had been a principal in AISD for 3 
years or less. 

The ACME project designed and used to provide professional development to help campus 
leaders support teachers implementing standards-based curriculum resources, however none were 
held in ACME’s third year. The effectiveness of professional development for campus 
administrators appears to depend on principal’s knowledge of systemic reform and readiness to 
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implement standards-based curriculum and instruction on their campuses as well as on support from 
central office leaders. 

Support from Stakeholders 

Stakeholders in the ACME project include elementary and middle school mathematics 
teachers, principals, central office administrators, as well as, parents, professionals in higher 
education, and other community members. In general, teachers supported the instructional practices 
of the ACME initiative in mathematics education highly; for example, a majority (90%) of the 
teachers surveyed on the LSC Teacher Questionnaire considered developing students’ conceptual 
understanding in mathematics and hands-on activities “very important.” A small proportion of 
teachers expressed opposition to implementing standards-based instruction by supplementing the 
curriculum resources with materials that are not standards-based. Opposition from the teachers’ 
union to implementing the curriculum resources surfaced in the Spring of 2000 but was incited 
primarily by teachers on one campus. The number of teachers not attending the second summer 
institute raises concern that the design of ACME professional development is not meeting their 
needs. 

According to teachers who responded to the LSC Teacher Questionnaire, parents continued 
to express neither strong support nor opposition to standards-based mathematics instruction as in 
previous years. District and ACME leaders have responded to opposition from vocal parents, 
however. To educate parents about what to expect from standards-based mathematics curriculum 
and instruction, many campuses have held family math nights annually, often with the support of 
ACME facilitators. ACME staff have also developed pamphlets to inform parents and distributed 
videos about standards-based mathematics. A new district initiative to spur parental involvement 
may further gamer parental support in the 2000-2001 school year. 

CONSISTENCY OF DISTRICT INITIATIVES 

The consistency of district initiatives has gone far to align district policy and practices with 
the ACME vision for mathematics education. The AISD Language and Literacy Department has 
been implementing the Balanced Literacy Program and the Science and Health Education 
Department has been implementing FOSS for several years. Both initiatives are based on a 
constructivist approach to teaching and learning. 

The new superintendent contracted with the Institute for Learning (IFL) in Pittsburgh to help 
district and campus leaders refocus teaching and learning districtwide. District staff and campus 
administrators have participated in workshops, demonstrations, and discussions with IFL staff. The 
district chose to focus on two of nine Principles of Learning, clear expectations and accountable talk, 
which ACME facilitators have posted and discussed in ACME professional development. While the 
knowledge and beliefs advocated by IFL appear to align with the ACME vision for mathematics 
education, it is unclear whether district and campus administrators are making connections explicit. 
IFL has the potential to help campus administrators become strong instructional leaders. This 
initiative could support the goals of the ACME project if the message about the connections is clear. 

Another local initiative has the potential to support the ACME vision for mathematics 
education, although in practice the support has been spotty. In the 1999-2000 school year, the district 
initiated the Account for Learning (AFL) funding source to improve instruction on 42 campuses 
where student achievement was low. The initiative included an instructional specialist on each of 
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these 42 campuses to support teachers. ACME staff were formative to the professional development 
for these instructional specialists and shared information about standards-based mathematics 
instruction. 

These specialists could participate in cognitive coaching, mentoring, and teacher 
collaboration necessary to help teachers develop standards-based pedagogical skills. However, only 
about five of the 42 specialists hired had participated in the ACME teacher cadre and had 
competence in standards-based mathematics instruction. Other AFL specialists were strong in 
language arts and some were pulled from classrooms to meet other organizational needs. Thus, a 
small number had the competence to lead standards-based mathematics instruction on their 
campuses. Moreover, interviews with specialists revealed that much of their time was spent 
mentoring new teachers, helping teachers analyze TAAS data, sharing strategies for TAAS 
preparation, and organizing campus instructional materials. To support the ACME vision for 
mathematics education, instructional specialists would be central to a plan to help teachers become 
strong implementers of standards-based mathematics instruction, including cognitive coaching and 
content-focused collaboration. The professional development provided AFL specialists may prepare 
them for some of these responsibilities, their success may depend on their beginning the position 
with a high level of knowledge and pedagogical skills in standards-based mathematics instruction as 
well as strong leadership skills and district and campus support. 

Curriculum Resources 

In the Spring of 1999, AISD decided on dual textbook adoptions. The district chose to 
supplement the ACME curriculum resource Investigations in Number , Data , and Space with the 
traditional texts Math in My World for elementary schools and to supplement CMP with 
Mathematics : Applications and Connections for middle schools. A committee of teachers used a 
rubric that the Dana Center developed to evaluate curriculum resources. Although the two ACME 
resources were rated the highest, the district chose a dual adoption to fill in a few gaps in the TEKS 
standards, which vary by grade level, that emerged in Investigations and in CMP. 

The dual adoption sent mixed messages to teachers and administrators. While adopting a 
textbook to fill a few gaps in the TEKS and appease stakeholders who prefer a textbook, it sends 
mixed messages about AISD’s direction in. mathematics education. In interviews, some teachers 
expressed concern about others not implementing Investigations and CMP. In classroom 
observations, a few teachers used the textbooks for topics covered in Investigations and CMP . In 
AISD, dual adoption was a compromise that deterred the complete implementation of standards- 
based curriculum resources. 

Student Assessment 

A persistent deterrent to implementing standards-based mathematics curriculum and 
instruction was teacher concern about the statewide assessment TAAS and preparing students to pass 
the test (see “Student TAAS Mathematics Results and the Quality of Teacher Implementation,” pp. 
5-8). As in previous years, teachers expressed anxiety about the compatibility of standards-based 
curriculum and instruction with student achievement on TAAS (see Batchelder & Christian, 1999). 
One teacher stated, “We are all bound by TAAS; I don’t feel like Investigations leads us to TAAS.” 
The fear of low TAAS performance continued to influence decisions about curriculum. One teacher 
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reported in April of 2000, “For the past six weeks, I have had to abandon Investigations to teach 
TAAS test-taking strategies.” 

AISD and the ACME project have taken several approaches to allay this anxiety. Early on, 
the ACME project addressed these teacher concerns by designing ACME professional development 
activities to examine TAAS items as they relate to standards-based curriculum and instruction. In the 
1999-2000 school year, AISD administrators established the policy that teachers would teach the 
state standards TEKS. The district also contracted the Dana Center’s professional development 
“TEKS for Leaders” for campus administrators and district curriculum staff. These sessions 
demonstrated the direct link between the TEKS and the TAAS. 

Institutionalization of ACME Reforms 

High Quality Professional Development 

The foundation for institutionalizing ACME reforms rests on the extensive, in-house 
professional development program that helps teachers learn to implement standards-based curriculum 
resources and instruction. If AISD decides to continue providing ACME professional development, 
staff development days, and stipends for teachers, many AISD teachers will continue to learn how to 
implement standards-based curriculum and instruction. This sustenance also depends on maintaining 
a small staff of high quality professional development facilitators. However, limitations on the 
quality of implementation most likely will persist without widely available structures of professional 
development that promote improvements in teachers’ pedagogical skills and content knowledge (e.g., 
cognitive coaching, content-focused collaborative inquiry, and mentoring). 

Support for Standards-Based Mathematics Education 

The strongest support for standards-based mathematics education currently comes from 
teachers and some district and campus administrators. Given the link between student mathematics 
achievement and strong implementation of standards-based instruction, an advantage of the ACME 
reforms is the impact on student learning. Thus, institutionalizing standards-based mathematics 
curriculum and instruction would support the central goal of AISD, improving student learning. To 
institutionalize the ACME reforms, work is still needed to inform district and campus administrators 
about standards-based instruction and the process of systemwide change and to gamer the support of 
a majority. A clear message about the direction of AISD mathematics education is lacking. 
Continued work educating parents about standards-based mathematics instruction and helping them 
feel comfortable with the changes is also necessary. Developing relationships with institutions of 
higher education could be a means for addressing the preparation of new hires in standards-based 
instruction and for improving the mathematics content knowledge of teachers. 
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Summary and Recommendations 

Strengths of ACME Project 

In the third year of the project, the ACME project presented the following strengths: 

• Strong implementation of standards-based mathematics curriculum and instruction was 
associated with high student achievement. 

• ACME professional development helped teachers learn to implement standards-based 
curriculum resources. 

• In conjunction with the ACME project, AISD provided all teachers with standards-based 
curriculum resources (including kits, copies of student sheets, and planning tools). 

Adaptations to ACME Project 

From the start, staff adapted ACME professional development to meet teachers’ needs by: 

• Focusing conversations and professional development activities on student thinking; 

• Developing the culture of a learning community; 

• Providing copies of student sheets and bilingual materials; 

• Designing separate sessions for special education teachers; 

• Establishing norms for professional development; 

• Integrating planning time into ACME professional development; 

• Developing planning tools to support implementation; and 

• Scheduling sessions on Saturday, after school, and at North and South locations. 

Although staff have adapted the ACME project to meet the needs of many teachers, some 

weaknesses in the design have not been addressed either by ACME or AISD. Districtwide structures 
that support implementation of standards-based instruction on all AISD campuses and meaningful 
teacher collaboration have not been developed. Teacher leadership from “experts” in standards- 
based instruction has remained untapped, except at a few sites. 

Challenges of ACME Project 

In the third year of the project, the ACME project manifested the following challenges: 

• Teachers across the district did not receive support for developing standards-based 
pedagogical skills and for deepening their mathematics content knowledge. 

• Low attendance at summer institutes indicated that ACME professional development was not 
a high priority for many teachers. 

• District and campus administrators did not uniformly support teacher implementation. 
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Recommendations 

1. Enlist district administrators to communicate a clear message about the district's vision for 
mathematics education because mixed messages have fostered piecemeal implementation of 
standards-based instruction across the district. Broadcast the message on the AISD cable 
channel to reach teachers, campus administrators, parents, and community members. In area 
principal meetings, include 10 minute updates on the mathematics program (e.g., attendance 
at ACME professional development, TEKS and TAAS mathematics objectives, and the 
association between standards-based instruction and student achievement). 

2. Make explicit the connections between ACME and other district initiatives , especially IFL f 
because the approaches to teaching and learning are compatible. IFL is an opportunity to 
strengthen the instructional leadership of district and campus administrators, which is a weak 
link in AISD’s implementation of standards-based mathematics. Making the connections 
explicit should foster a shared vision for AISD’s direction in curriculum and instruction and 
bolster necessary administrative support. Strong principal support occurs when 
administrators have knowledge of standards-based instruction and the process of systemic 
reform, commit and advocate for implementation, and organize teacher collaboration and 
leadership (Batchelder & Christian, 1999; St. John et al., 1999). If AISD is not able to 
bolster administrative support for standards-based mathematics instruction, it should look at 
other mathematics programs. 

3. Hire and train campus instructional specialists who are skilled in standards-based 
mathematics instruction through AFL funding. Establish collaborative relationships between 
these specialists and ACME facilitators to provide a network of strong support for 
implementation on campuses. Concentrate this campus support on cognitive coaching and 
content-focused collaboration. By developing effective forms of campus support, AISD will 
help more teachers become strong implementers of standards-based mathematics instruction, 
which is linked to high levels of student achievement on TAAS mathematics (especially 
problem-solving skills that will be key to passing future versions of TAAS). 

4. Provide new ACME staff with professional development to maintain the quality of ACME 
professional development for teachers. To ease the transition in ACME staff, develop 
cognitive coaching among team members and routinely examine teacher evaluations of 
ACME professional development to devise strategies to improve facilitators* skills. 
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Appendix A. Gains and Losses in Student TAAS Mathematics 

Figure 21 presents the gains and losses in the percentage of students passing TAAS 
mathematics between the 1998-99 and 1999-2000 school years by grade levels and by disaggregated 
groups (i.e., all students, African American, Hispanic, White, and economically disadvantaged). 
This figure shows that the greatest gains were made by African American, Hispanic, and 
economically disadvantaged students (except for 3 rd grade students), although their percentage 
passing continued to lag behind White students (see Figures 1 through 12). 

Figure 22 presents the gains and losses in the gains and losses in the mean TLI between the 
1998-99 and 1999-2000 school years by grade levels and by disaggregated groups. This figure also 
demonstrates that greatest gains were made by African American, Hispanic, and economically 
disadvantaged students than by White students, although the mean TLI for these groups was 
consistently lower than that of White students (see Figures 1 through 12). 

Figure 21. Gains and Losses in Percentage of Students Passing TAAS Mathematics Between 

1998-99 and 1999-2000 




All Students African American Hispanic White Economically 

Disadvantaged 



H Grade 3 U Grade 4 □ Grade 5 □ Grade 6 fl Grade 7 El Grade 8 



Figure 22. Gains and Losses in Mean TLI for Students in TAAS Mathematics Between 1998- 

99 and 1999-2000 
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Appendix B. ACME Professional Development and Change in Implementation 

Table 1. Frequencies of Changes in the Number of Professional Development Days by Changes 
in Observation Ratings from Spring of 1999 to Spring of 2000. 
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1999-2000 Local Systemic Change 
Pre-Classroom Observation Interview 



After you have expressed appreciation to the teacher for allowing you to observe the class, ask 
the following question: 

1 . What has this class been doing in mathematics/science recently? 

PROBES: What unit are you working on? 

What instructional materials are you using ? 

2. What do you anticipate doing in your mathematics/science class on the day I will be 
observing? 

PROBE: What do you hope students will learn as a result of the work you have 

planned? 

3. What is the next step for this class? 

4. Is there anything in particular that I should know about the group of students that I will be 
observing? 



Note that the evaluator will need to be thoroughly conversant with the instructional materials designated for use by the LSC in order to complete the 
observation ratings. 
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1999-2000 Local Systemic Change 
Post-Classroom Observation Interview 



After you have expressed appreciation to the teacher for allowing you to observe the class, ask 
the following questions: 

1 . Were there any ways in which the lesson was different from what you had planned? 

2. What did this lesson tell you about what your students are learning and still need to leam 
in mathematics/science? 

PROBE: How do you plan to further assess the students’ learning? 

3 . What challenges have you faced in encouraging your students to be actively engaged in 
this mathematics/science class? 

PROBE: How have you approached these challenges? 

4. What is the next step for this class? 
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NOTE: 



This form is included for information purposes only. Evaluators will need to 
complete the form on the Web. 



1999-2000 Local Systemic Change 
Classroom Observation Protocol 1 



Background Information 

Project 

LSC ID 2 

Subject Observed 3 

Grade Level 



Date of Observation 

Time of Observation: 

Start _ End 

Observer 

Observer’s Role in Project: 

Lead Evaluator 

Other Certified Observer 



Section One: Contextual Background and Activities 

In this section, please fill in the circles that best describe the class. For each item , be sure to fill in 
all responses that apply. 



I. Classroom Demographics and Context 

A. What is the total number of students in 
the class at the time of the observation? 

O 1 5 or fewer 
O 1630 
O 2135 
O 2630 
O 3 1 or more 



B. What is the approximate percentage of 
white (not Hispanic origin) students in 
this class? 

O 030 percent 
O 1 135 percent 
O 2630 percent 
O 5135 percent 
O 76300 percent 



C. Indicate the teacher $: 

1 . Gender 

O Male O Female 

2. Race/Ethnicity 

O African-American (not Hispanic origin) 
O American Indian or Alaskan Native 
O Asian or Pacific Islander 
O Hispanic 

O White (not Hispanic origin) 

O Other 



D. If applicable, indicate the teacher aideS : 

1. Gender 

O Male O Female 

2. Race/Ethnicity 

O African-American (not Hispanic origin) 
O American Indian or Alaskan Native 
O Asian or Pacific Islander 
O Hispanic 

O White (not Hispanic origin) 

O Other 



Be sure you have read the “1999-2000 Local Systemic Change Classroom Observations: Guidelines for Evaluators” and have completed the “Pre- 



2 Use the LSC ID number as indicated in the Classroom Observation Sample provided by HR1. 

3 In mathematics/science projects observe the subject for which the teacher was sampled. 
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E. Rate the adequacy of the physical environment. 



1 . Classroom resources: 



o o 


o 


O 


O 


1 2 


3 


4 


5 


Sparsely equipped 






Rich in resources 


2. Classroom Space: 


o o 


O 


O 


o 


1 2 
Crowded 


3 


4 


5 

Adequate space 


3. Room arrangement: 


o o 


O 


O 


o 


1 2 
Inhibited interactions 
among students 


3 


4 


5 

Facilitated interactions 
among students 



II. Lesson Description 

In a paragraph or two, describe the lesson you observed. Include where this lesson fits in the overall unit 
of study. Be sure to include enough detail to provide a context for your ratings of this lesson and also to 
allow you to recall the details of this lesson when needed in future years for longitudinal analysis. 



III. Purposes of Lesson 

A. Indicate the major 4 content area(s) of this lesson or activity. 



o 


1 . 


Numeration and number theory 


O 16. 


Life Science 


o 


2. 


Computation 




(please specify: 


o 


3. 


Estimation 


O 17. 


Physical science 


o 


4. 


Measurement . 




(please specify: 


o 


5. 


Patterns and relationships 


O 18. 


Earth/space sciences 


o 


6. 


Pre-algebra 


O 


a. Astronomy 


o 


7. 


Algebra 


O 


b. Oceanography 


o 


8. 


Geometry and spatial sense 


O 


c. Geology 


o 


9. 


Functions (including trigonometric 


O 


d. Meteorology 






functions) and pre-calculus concept 


O 


e. Environmental sciences 


o 


10. 


Data collection and analysis 


O 19. 


Engineering and design principles 


o 


11. 


Probability 


O 20. 


History of mathematics/science 


o 


12. 


Statistics (e.g., hypothesis tests, 










curve-fitting, and regression) 


O 21. 


None of the above (please explain) 


o 


13. 


Topics from discrete mathematics 










(e.g., combinatorics, graph theory, 










recursion) 






o 


14. 


Mathematical structures (e.g., vector spaces, 










groups, rings, fields) 






o 


15. 


Calculus 





“Major” means was used or addressed for a substantial portion of the lesson; if you were describing the lesson to someone, this feature would help 
characterize it. 



o 
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B. Indicate the primary intended purpose(s) of this lesson or activity based on the pre- and/or post- 
observation interviews with the teacher. 

O 1 . Identifying prior student knowledge 

O 2. Introducing new concepts 

O 3. Developing conceptual understanding 

O 4. Reviewing mathematics/science concepts 

O 5. Developing problem-solving skills 

O 6. Learning mathematics/science processes, algorithms, or procedures 

O 7. Learning vocabulary/specific facts 

O 8. Practicing computation for mastery 

O 9. Developing appreciation for core ideas in mathematics/science 

O 10. Developing students ’awareness of contributions of scientists/mathematicians of diverse 
backgrounds 

O 11. Assessing student understanding 

IV. Instructional Materials 

A. Is this lesson based on instructional materials designated for use by this LSC? 

O Yes O No, SKIP to Part V below 



B. Indicate the single set of LSC-designated instructional materials intended to form the basis of this 
lesson (e.g., FOSS; Insights; STC; Investigations in Number, Data, and Space; Connected Math; 
IMP; SEPUP), based on the information provided in the pre-observation interview. 

Please specify. 

C. How closely did the lesson adhere to the instructions provided in the teacher’s manual? 

O Exactly, SKIP to Part V below 

O Almost totally O Mostly O Somewhat O A little O Hardly at all 

D. How did the adaptations affect the quality of the lesson? 

O Helped a lot O Helped a little O Neutral O Hurt a little O Hurt a lot 



V. Classroom Instruction 

A. Indicate the major 5 way(s) in which student activities were structured. 



“Major” means was used or addressed for a substantial portion of the lesson; if you were describing the lesson to someone, this feature would help 
characterize it. 



O As a whole group O As small groups O As pairs O As individuals 



B. Indicate the major ? way(s) in which students engaged in class activities. 



O Entire class was engaged in the same activities at the same time. 

O Groups of students were engaged in different activities at the same time (e.g., centers). 
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C. Indicate the major 6 activities of students in this lesson. When choosing an Umbrella’bategory, be sure to 
indicate subcategories that apply as well. (For example, if you mark listened to a presentation, ’indicate by 
whom.) 

O 1 . Listened to a presentation: 

O a. By teacher (would include: demonstrations, lectures, media presentations, extensive procedural instructions) 

O b. By student (would include informal, as well as formal, presentations of their work) 

O c. By guest speaker/bxpert’fcerving as a resource 

O 2. Engaged in discussion/seminar: 

O a. Whole group' 

O b. Small groups/pairs 

O 3. Engaged in problem solving/investigation: 

O a. Worked with manipulatives 
O b. Played a game to build or review knowledge/skills 
O c. Followed specific instructions in an investigation 
O d. Had some latitude in designing an investigation 

O e. Recorded, represented and/or analyzed data 
O f. Recognized patterns, cycles or trends 
O g. Evaluated the validity of arguments or claims 
O h. Provided an informal justification or formal proof 

O 4. Engaged in reading/reflection/written communication about mathematics or science: 

O a. Read about mathematics/science 
O b. Answered textbook/worksheet questions 

O c. Reflected on readings, activities, or problems individually or in groups 
O d. Prepared a written report 

O e. Wrote a description of a plan, procedure, or problem-solving process 
O f. Wrote reflections in a notebook or journal 

O 5. Used technology/audio-visual resource: 

O a. To develop conceptual understanding 
O b. To learn or practice a skill 
O c. To collect data (e.g., probeware) 

O d. As an analytic tool (e.g., spreadsheets or data analysis) 

O e. As a presentation tool 

O f. For word processing or as a communications tool (e.g., e-mail, Internet, Web) 

O 6. Other activities 

O a. Arts and crafts activity 
O b. Listened to a story 
O c. Wrote a poem or story * 

O d. Other (Please specify.) 




“Major” means was used or addressed for a substantial portion of the lesson; if you were describing the lesson to someone, this feature would help 
characterize it. 
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D. Comments 

Please provide any additional information you consider necessary to capture the activities or context of 
this lesson. Include comments on any feature of the class that is so salient that you need to get it bn the 
table’hght away to help explain your ratings; for example, the class was interrupted by a fire drill, the kids 
were excited about an upcoming school event, or the teacher^ tone was so warm (or so hostile) that it was 
an overwhelmingly important feature of the lesson. 



Section Two: Ratings 

In Section One of this form, you documented what occurred in the lesson. In this section, you are 
asked to rate each of a number of key indicators in four different categories, from 1 (not at all) to 5 
(to a great extent). You may list any additional indicators you consider important in capturing the 
essence of this lesson and rate these as well. Use your “Ratings of Key Indicators” (Part A) to inform 
your “Synthesis Ratings” (Part B). It is important to indicate in “Supporting Evidence for Synthesis 
Ratings” (Part C) what factors were most influential in determining your synthesis ratings and to give 
specific examples or quotes to illustrate those factors. 

Note that any one lesson is not likely to provide evidence for every single indicator; use 6, “Don't 
know” when there is not enough evidence for you to make a judgment. Use 7, “N/A” (Not 
Applicable) when you consider the indicator inappropriate given the purpose and context of the 
lesson. Section Two concludes with ratings of the likely impact of instruction, and a capsule 
description of the lesson. 
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I. Design 

A. Ratings of Key Indicators 

1. The design of the lesson incorporated tasks, roles, and 
interactions consistent with investigative mathematics/science. 

2. The design of the lesson reflected careful planning and 
organization. 

3. The instructional strategies and activities used in this 
lesson reflected attention to students ’experience, 
preparedness, and/or learning styles. 

4. The resources available in this lesson contributed to 
accomplishing the purposes of the instruction. 

5. The instructional strategies and activities reflected attention 
to issues of access, equity, and diversity for students 

(e.g., use of Wait time/booperative learning, language- 
appropriate strategies/materials). 

6. The design of the lesson encouraged a collaborative 
approach to learning. 

7. Adequate time and structure were provided for £ense-making.” 

8. Adequate time and structure were provided for wrap-up 
and closure. 

9. Formal assessments of students were consistent with 
investigative mathematics/science. 

10. Design for future instruction takes into account what 
transpired in the lesson. 



11 . 



Not 

at 

all 



To a 
great 
extent 



Dont 

know 



1 2 3 4 5 



1 2 3 4 5 



1 2 3 4 5 



1 2 3 4 5 



1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 



B. Synthesis Rating 



1 


2 


3 


4 


5 


Design of the lesson not at 
all reflective of best 
practice in 
mathematics/science 
education 








Design of the lesson 
extremely reflective of 
best practice in 
mathematics/science 
education 



C. Supporting Evidence for Synthesis Rating 




O 
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n. Implementation 



A. Ratings of Key Indicators 

1 . The instruction was consistent with the underlying 
approach of the instructional materials designated 
for use by the LSC. 

2. The instructional strategies were consistent with 
investigative mathematics/science. 

3. The teacher appeared confident in his/her ability to teach 
mathematics/science. 

4. The teacher^ classroom management style/strategies 
enhanced the quality of the lesson. 

5. The pace of the lesson was appropriate for the developmental 
levels/needs of the students and the purposes of the lesson. 



Not 

at 

all 



To a 

great Dont 
extent know N/A 



1 2 3 4 5' 6 7 

1 2 3 4 5 6 7 



1 2 3 4 5 



6 7 



1 2 3 4 5 



6 7 



1 2 3 4 5 



6 7 



6. The teacher took into account prior knowledge of students. 



2 3 4 5 



6 7 



7. The teacherS questioning strategies were likely to enhance the 
development of student conceptual understanding/problem solving 
(e.g., emphasized higher order questions, appropriately used 
Wait time, ’Identified prior conceptions and misconceptions). 



2 3 4 5 



6 7 



8. The lesson was modified as needed based on teacher 
questioning or other student assessments. 



2 3 4 5 



6 7 



9. 



2 3 4 5 



B. Synthesis Rating 



1 


2 


3 


4 


5 


Implementation of the 
lesson not at all reflective 
of best practice in 
mathematics/science 
education 








Implementation of the 
lesson extremely reflective 
of best practice in 
mathematics/science 
education 



C. Supporting Evidence for Synthesis Rating 
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HI. Mathematics/Science Content 



A. Ratings of Key Indicators 



Not 

at 

all 



To a 

great Dont 

N/A 

extent know mm 



1 . The mathematics/science content was significant and worthwhile. 



2 3 4 5 



6 7 



2. The mathematics/science content was appropriate for the 
developmental levels of the students in this class. 



2 3 4 5 



6 



7 



3. Students were intellectually engaged with important ideas 
relevant to the focus of the lesson. 



2 3 4 5 



6 



7 



4. Teacher-presented information was accurate. 



2 3 4 5 



5. The teacher displayed an understanding of mathematics/science 

concepts (e.g., in his/her dialogue with students). 1 2 3 4 5 

6. Mathematics/science was portrayed as a dynamic body of 
knowledge continually enriched by conjecture, investigation 

analysis, and/or proof/justification. ' 1 2 3 4 5 

7. Elements of mathematical/science abstraction (e.g., symbolic 
representations, theory building) were included when it was 

important to do so. 1 2 3 4 5 



8. Appropriate connections were made. to other areas of mathematics/ 
science, to other disciplines, and/or to real-world contexts. 



2 3 4 5 



9. The degree of "sense-making" of mathematics/science content 
within this lesson was appropriate for the developmental 

levels/needs of the students and the purposes of the lesson. 1 2 3 4 5 



6 7 

6 7 

6 7 

6 7 

6 7 

6 7 



10 . 



2 3 4 5 



B. Synthesis Rating 



1 


2 


3 


4 


5 


Mathematics/science 








Mathematics/science 


content of lesson not at all 








content of lesson 


reflective of current 








extremely reflective of 


standards for 








current standards for 


mathematics/science 








mathematics/science 


education 








education 



C. Supporting Evidence for Synthesis Rating 
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IV. Classroom Culture 



Al. Ratings of Key Indicators 

1. Active participation of all was encouraged and valued. 

2. There was a climate of respect for students ’ideas, 
questions, and contributions. 

3. Interactions reflected collaborative working relationships 
among students (e.g., students worked together, talked with 
each other about the lesson). 



Not 

at 

all 



To a 

great Dont 
extent know N/A 



1 2 3 4 5 



6 7 



1 2 3 4 5 



6 7 



1 2 3 4 5 



6 7 



4. Interactions reflected collaborative working relationships 
between teacher and students. 



2 3 4 5 



6 7 



5. The climate of the lesson encouraged students to generate 
ideas, questions, conjectures, and/or propositions. 

6. Intellectual rigor, constructive criticism, and the challenging 
of ideas were evident. 



1 2 3 4 5 



6 



1 2 3 4 5 



6 



7 



7 



7. 



2 3 4 5 



A2. Respect for Diversity 

Based on the culture of a classroom, observers are generally able to make inferences about the extent to 
which there is an appreciation of diversity among students (e.g., their gender, race/ethnicity, and/or cultural 
background). While direct evidence that reflects particular sensitivity or insensitivity toward diversity is not 
often observed, we would like you to document any examples you do see. If any examples were observed, 
please check here □ and describe below: 



B. Synthesis Rating 



1 


2 


3 


4 


5 1 


Classroom culture 
interfered with student 
learning 








Classroom culture 
facilitated the learning of 
all students ! 



C. Supporting Evidence for Synthesis Rating 
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V. Overall Ratings of the Lesson 

A. Likely Impact of Instruction on Students’ Understanding of Mathematics/Science 

While the impact of a single lesson may well be limited in scope, it is important to judge whether the lesson is 
likely to help move students in the desired direction. For this series of ratings, consider all available 
information (i.e., your previous ratings of design, implementation, content, and classroom culture, and the pre- 
and post-observation interviews with the teacher) as you assess the likely impact of this lesson. Feel free to 
elaborate on ratings with comments in the space provided. 

Select the response that best describes your overall assessment of the likely effect of this lesson in each of the 
following areas. 









Mixed or 














Negative 




neutral 




Positive 


Dont 








effect 




effect 




effect 


know 


N/A 


1 . 


Students ’understanding of mathematics/science as a dynamic 
body of knowledge generated and enriched by investigation. 


o 


o 


o 


o 


o 


o 


o 


2. 


Students ’understanding of important mathematics/science 
concepts. 


o 


o 


o 


o 


o 


o 


o 


3. 


Students ’capacity to carry out their own inquiries. 


o 


o 


o 


o 


o 


o 


o 


4. 


Students’ability to apply or generalize skills and concepts to 
other areas of mathematics/science, other disciplines, and/or 
real-life situations. 


o 


o 


o 


o 


o 


o 


o 


5. 


Students’self-confidence in doing mathematics/science. 


o 


o 


o 


o 


o 


o 


o 


6. 


Students’interest in and/or appreciation for the discipline. 


o 


o 


o 


o 


o 


o 


o 



Comments (optional): 
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B. Capsule Description of the Quality of the Lesson 

In this final rating of the lesson, consider all available information about the lesson, its context and 
purpose, and your own judgment of the relative importance of the ratings you have made. Select the 
capsule description that best characterizes the lesson you observed. Keep in mind that this rating is hot 
intended to be an average of all the previous ratings, but should encapsulate your overall assessment of the 
quality and likely impact of the lesson. Please provide a brief rationale for your final capsule description 
of the lesson in the space provided. 

O Level 1: Ineffective Instruction 

There is little or no evidence of student thinking or engagement with important ideas of 
mathematics/science. Instruction is unlikely to enhance students’ understanding of the discipline or 
to develop their capacity to successfully “do” mathematics/science. Lesson was characterized by 
either (select one below): 

O Passive “Learning” 

Instruction is pedantic and uninspiring. Students are passive recipients of information from the 
teacher or textbook; material is presented in a way that is inaccessible to many of the students. 

O Activity for Activity’s Sake 

Students are involved in hands-on activities or other individual or group work, but it appears to 
be activity for activity’s sake. Lesson lacks a clear sense of purpose and/or a clear link to 
conceptual development. 

O Level 2: Elements of Effective Instruction 

Instruction contains some elements of effective practice, but there are substantial problems in the 
design, implementation, content, and/or appropriateness for many students in the class. For example, 
the content may lack importance and/or appropriateness; instruction may not successfully address the 
difficulties that many students are experiencing, etc. Overall, the lesson is quite limited in its 
likelihood to enhance students’ understanding of the discipline or to develop their capacity to 
successfully “do” mathematics/science. 

O Level 3: Beginning Stages of Effective Instruction (Select one below.) 

O Low 3 O Solid 3 O High 3 

Instruction is purposeful and characterized by quite a few elements of effective practice. Students are, 
at times, engaged in meaningful work, but there are some weaknesses in the design, implementation, 
or content of instruction. For example, the teacher may short-circuit a planned exploration by telling 
students what they “should have found”; instruction may not adequately address the needs of a 
number of students; or the classroom culture may limit the accessibility or effectiveness of the lesson. 
Overall, the lesson is somewhat limited in its likelihood to enhance students’ understanding of the 
discipline or to develop their capacity to successfully “do” mathematics/science. 

O Level 4: Accomplished, Effective Instruction 

Instruction is purposeful and engaging for most students. Students actively participate in 
meaningful work (e.g., investigations, teacher presentations, discussions with each other or the 
teacher, reading). The lesson is well-designed and the teacher implements it well, but adaptation of 
content or pedagogy in response to student needs and interests is limited. Instruction is quite likely to 
enhance most students’ understanding of the discipline and to develop their capacity to successfully do” 
mathematics/science. 

O Level 5: Exemplary Instruction 

Instruction is purposeful and all students are highly engaged most or all of the time in meaningful 
work (e.g., investigation, teacher presentations, discussions with each other or the teacher, reading). 
The lesson is well-designed and artfully implemented, with flexibility and responsiveness to students’ 
needs and interests. Instruction is highly likely to enhance most students' understanding of the 
discipline and to develop their capacity to successfully do’hiathematic s/science. 

Please provide your rationale for the capsule rating: 
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Local Systemic Change 
Pre-Observation Interview with 
Professional Development Facilitator 



1 . Please talk with me briefly about the primary purposes of the professional development 
session I will be observing. 

PROBE: What do you hope participants will gain as a result of their participation 

in this session? 1 

2. What do you anticipate happening during the session I will be observing? 

PROBES: Will the session include any of the materials the LSC has designated for classroom 
use? 

If so, how will they be used? 

3. How does this session fit into the sequence of professional development experiences planned 
for this district’s teachers? 

PROBES: What experiences have these participants had with the LSC prior to this 
session? 

What will they do next, with regard to professional development? 

4. Tell me a little about your background as it relates to the session you will be facilitating. 

5. Is there anything in particular that I should know about the participants who will be attending 
this session? 



1 Several of the ratings on the Professional Development Observation Protocol require an understanding of the 
intended purposes of the session. If the facilitator is not explicit in describing the purposes of the session, further 
probes may be needed. Additional probes might include direct questions about the extent to which the session is 
intended to enhance participants’ content knowledge, to explore pedagogical strategies/instructional materials or to 
explore strategies/issues/roles for teacher leaders, principals, or others in leadership positions. Refer to Section 
One, EIIA on the Professional Development Observation Protocol for a list of potential purposes. 
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NOTE: 



This form is included for information purposes only. Evaluators will need to 
complete the form on the Web. 



1999-2000 Local Systemic Change 
Professional Development Observation Protocol 1 



Background Information 



Project 

Location 

Observer 

Observer's Role in Project: O Lead Evaluator 



Date of Observation 

If you are submitting two professional development observations 
for this date , indicate whether this was the first or second 
session observed. O 1 st O 2 nd 

Approximate Duration of Observation 2 : 

O 1 hour O 3 hours 

O 2 hours O half day 

O Other 



Subject Targeted by session O Mathematics O Science O Both Mathematics and Science 



Section One: Contextual Background And Activities 

In this section, please fill in the circles that best describe the session. For each item, be sure to fill in all 
responses that apply. 

I. Session Demographics 

A. What is the total number of participants attending this session? 

O 1-5 O 6-10 O 11-20 O 21-50 O 51-100 O More than 100 

B. Please describe the targeted subject(s)/grade level(s)/audience for this professional development session. 

1 . This session was intended to improve the teaching of: (select all that apply) 

O Elementary science O Elementary mathematics 

O Middle grades science O Middle grades mathematics 

O High school science O High school mathematics 

2. Participants were: 

O Lead teachers for the LSC projects 
O Other (non-lead) teachers 
O Administrators 

O Other (Please specify.) 



1 Be sure you have read the “1999-2000 Local Systemic Change Professional Development Observations: Guidelines for Evaluators” and have completed 
the “Pre -Observation Interview with Professional Development Facilitator” before observing the session. 



The observation recorded on this form should be no less than one hour and no more than half a day. 
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C. Please describe the major presenters/facilitators 3 for this particular one-hour to half-day 
professional development session. 



1 . 



Indicate the number of presenters/facilitators in each gender and race/ethnicity category. 





African-American 
(not Hispanic-origin) 


American Indian 
or Alaskan Native 


Asian or 
Pacific Islander 


Hispanic 


White (not 
Hispanic origin) 


Other 


Male 














Female 















2 . 



Indicate the number of presenters/facilitators for this particular session with each affiliation. 



Regular 

Full-Time 

or 

Part-Time 

Classroom 

Teachers 


Teachers 

on 

Special 

Assignment 4 


District 

Mathematics/ 

Science 

Supervisor 


Other 

District 

Personnel 


University 

Mathematics/ 

Science 

Faculty 


University 

Mathematics/ 

Science 

Education 

Faculty 


Business 

Industry 

Mathematicians/ 

Scientists 


Other 

Non- 

District 

Personnel 



















II. Session Context 

In a few sentences, describe the session you observed. Include: (a) whether the observation covered a partial or 
complete session, (b) whether there were multiple break-out sessions, and (c) where this session fits in the project’s 
sequence of professional development for those in attendance. 

in. Session Focus 

A. Indicate the primary intended purpose(s) of this professional development session based on the information 
provided by the project staff or session organizer/facilitator. 

O 1. Increasing mathematics/science content knowledge of participants. {Be sure to complete Category III: 
Mathematics /Science Content and Category VILA: Likely Impact on Participants ' Capacity to Provide 
High-Quality Mathematics/Science Education , in Section Two of the protocol) 

O 2. Explicit attention to classroom pedagogy/designated instructional materials. {Be sure to complete Category 
IV: Exploring Pedagogy /Instructional Materials and Category VILA: Likely Impact on Participants' 
Capacity to Provide High-Quality Mathematics/Science Education , in Section Two of the protocol) 

O a. Creating a vision of effective mathematics/science instruction 
O b. Understanding student thinking/leaming about mathematics/science content 
O c. Learning how to use specific instructional materials in the classroom 
O d. Learning how to use technology in the classroom. 

O e. Learning pedagogical/classroom management strategies 
O f. Considering issues of access, equity, and diversity 
O g. Designing or scoring student assessments 

O h. Considering issues of scope and sequence (e.g., K-12 curricular frameworks) 

O 3. Explicit attention to strategies/issues/roles of teacher leaders, principals, or others in leadership positions. 
{Be sure to complete Category V: Leadership Content and Category VII.B: Likely Impact on Participants * 
Leadership Capacity , in Section Two of the protocol) 

O 4. Other major purposes: 

O a. Orientation to the project 

O b. Assessing participants’ knowledge/skills 

O c. Building professional networks among educators 

O d. Promoting/exploring reflective practice 

O e. Developing the capacity of participants to use technology 

O f. Involving administrators and/or other school/district personnel in the reform process 



3 In some instances this may not be appropriate, e.g., a session in which a group of teachers meets after school to discuss their action research projects may have no 
presenters or facilitators. In these instances, please leave the presenteis/facilitators cells blank. 

4 Defined as teachers released full-time from classroom responsibilities to work on assignments such as the LSC project. 
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B. Indicate the mayor 5 mathematics/science content area(s) addressed in this professional development session, whether 
increasing content knowledge was a stated purpose or the mathematics/science content was simply a vehicle for 
achieving other purposes. 



o 


1 . 


Numeration and number theory 


o 


16. 


l ife Science ("Please specifv.l 


o 


2. 


Computation 


o 


17. 


Physical science (Please specifv.) 


o 


3. 


Estimation 


o 


18. 


Earth/space sciences 


o 


4. 


Measurement 






O a. Astronomy 


o 


5. 


Patterns and relationships 






O b. Oceanography 


o 


6. 


Pre-algebra 






O c. Geology 


o 


7. 


Algebra 






O d. Meteorology 


o 


8. 


Geometry and spatial sense 






O e. Environmental science 


o 


9. 


Functions (including trigonometric 


o 


19. 


Engineering and design principles 






functions) and pre-calculus 


o 


20. 


History of mathematics/science 






concepts 


o 


21. 


Mathematics/science as a way of knowing 


o 


10. 


Data collection and analysis 






(e.g., inquiry, problem solving) 


o 


11. 


Probability 








o 


12. 


Statistics (e.g., hypothesis tests, 












curve-fitting, and regression) 


□ 


Mathematics/science concepts were not included 


o 


13. 


Topics from discrete mathematics 




as either an explicit focus or a vehicle for achieving other 






(e.g., combinatorics, graph theory, 




professional development purposes 






recursion) 








o 


14. 


Mathematical structures (e.g., 












vector spaces, groups, rings, 












fields) 








o 


15. 


Calculus 









IV. Professional Development Activities 

A. Were any of the instructional materials intended for classroom use as part of the LSC (e.g., FOSS; Insights; STC; 
SEPUP; Investigations in Number, Data, and Space; Connected Math; IMP; Core Plus) a focus of the professional 
development session? 



O No 

O Yes Please specify. 



B. Indicate the major 5 activities of participants in this session. When choosing an "umbrella" category, be sure to indicate 
subcategories that apply as well. For example, if you mark "formal presentations," indicate by whom. 



as: 



O 1 . Listened to a formal presentation by: O 2. Engaged in discussions/seminars/reporting out structured 



O a. Session presenter/facilitator 
O b. Participant(s) 



O a. Entire group led by presenter/facilitator 
O b. Entire group led by participant(s) 

O c. Subsets of the group 



O 3. Engaged in problem solving/investigation focusing on disciplinary content, pedagogy, and/or reform issues 
O 4. Read about disciplinary content, pedagogy, or reform issues 
O 5. Wrote about disciplinary content, pedagogy, or reform issues 



67 

5 “Major” means was used or addressed for a substantial portion of the session; if you were describing the session to someone, this feature would help characterize it. 
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C. Indicate the major professional development approaches used in this session . 6 

O W orkshop/institute/course/seminar 
O Receiving formal professional development via technology 
O Study groups/“kit clubs”/discussion groups/school-based meetings 
O Coaching/mentoring 

O Other: 

D. Comments 

Please provide any additional information you consider necessary to capture the activities or context of this professional 
development session. Include comments on any feature of the session that is so salient that you need to get it "on the table” 
right away to help explain your ratings. 



Section Two: Ratings 

In Section One of this form, you documented what occurred in the session. In this section, you are asked to use that 
information, as well as any other pertinent observations, to rate each of a number of key indicators in six different categories, 
from 1 (not at all) to 5 (to a great extent). 

Note that any one session is not likely to provide evidence for every single indicator; use 6, “Don't know” when there is not 
enough evidence for you to make a judgment. Use 7, “N/A” (Not Applicable) when you consider the indicator inappropriate 
given the purpose and context of the session. For example, a session that focuses on engaging teachers in mathematics/science 
inquiry may choose not to address classroom applications. In that case, key indicator #8 under Category I (Design), “The design 
of the session provided opportunities for teachers to consider classroom applications of resources, strategies, and techniques,” 
would be rated “N/A,” rather than “not at all.” 

Similarly, there may be entire rating categories that are not applicable to a particular session. For example, categories III, IV, 
and V (Content) and Overall Ratings VIIA (Likely Impact on Participants’ Capacity to Provide High Quality 
Mathematics /Science Education) and VIIB (Likely Impact on Participants’ Leadership Capacity) each have a box to check when 
the entire rating category is judged to be inappropriate for the session 7 . Categories I (Design), II (Implementation), and VI 
(Culture of the Professional Development Session) are ones in which specific indicators may be “not applicable,” but the overall 
category should routinely be rated for any observation. 

Note that you may list any additional indicators you consider important in capturing the essence of this session and rate these as 
well. 

Use your “Ratings of Key Indicators” (Part A) to inform your “Synthesis Ratings” (Part B). It is important to indicate in 
“Supporting Evidence for Synthesis Ratings” (Part C) what factors were most influential in determining your synthesis ratings 
and to give specific examples or quotes to illustrate those factors. Section Two concludes with ratings of the likely impact of 
professional development, and a capsule description of the session. 




Observers should refer to the Annotated Guide to the Professional Development Observation Protocol for descriptions of each of these professional development 



approaches. 
7 T 



In most cases, the categories you rate will be consistent with the purposes marked in Section One. Part IU.A. 1 through 3. 
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Not 








To a 




I. Design 


at 








great 


Don’t 






all 






extent 


know 


A. Ratings of Key Indicators 














1 . 


The design of the session incorporated tasks, roles, and 














- 


interactions consistent with a spirit of investigation. 


1 


2 


3 


4 


5 


6 


2. 


The instructional strategies and activities used in this 
















session reflected attention to participants’ experience, 
















preparedness, and/or learning styles. 


1 


2 


3 


4 


5 


6 


3. 


The session effectively built on participants’ knowledge of 
















content, teaching, learning, and/or the reform process. 


1 


2 


3 


4 


5 


6 


4. 


The strategies in this session were appropriate for accomplishing 
















the purposes of the LSC professional development. 


1 


2 


3 


4 


5 


6 


5. 


The design of the session reflected careful planning and 
















organization. 


1 


2 


3 


4 


5 


6 


6. 


The design of the session included “framing” the activity 
















to help participants understand the purpose of the session and 
















where it fits into the larger professional development picture. 


1 


2 


3 


4 


5 


6 


7. 


The design of the session encouraged a collaborative 
















approach to learning. 


1 


2 


3 


4 


5 


6 


8. 


The design of the session provided opportunities for teachers 
















to consider classroom applications of resources, strategies, 
















and techniques. 


1 


2 


3 


4 


5 


6 


9. 


Adequate time and structure were provided for “sense-making,” 
















including reflection about concepts, strategies, issues, etc. 


1 


2 


3 


4 


5 


6 


10. 


Adequate time and structure were provided for participants 
















to share experiences and insights. 


1 


2 


3 


4 


5 


6 


11. 


Adequate time and structure were provided for wrap-up and closure. 


1 


2 


3 


4 


5 


6 


12. 




1 


2 


3 


4 


5 





N/A 



7 



7 

7 

7 



7 



7 

7 



7 

7 

7 

7 



B. Synthesis Rating 



1 ‘ 


2 


3 


4 


5 


Design of the session not 
at all reflective of best 
practice for professional 
development 








Design of the session 
extremely reflective of 
best practice for 
professional development. 



C. Supporting Evidence for Synthesis Rating 
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II. Implementation 

A. Ratings of Key Indicators 

1. Formal presentation(s) included in the session were carried 
out effectively. 

2. The facilitators)’ contributions during the course of the session 
enhanced the quality of the session. 

3. The facilitator(s) effectively modeled questioning strategies that are 
likely to enhance the development of conceptual understanding 
(e.g., emphasis on higher-order questions, appropriate use of 
“wait time,” identifying prior conceptions and misconceptions.) 

4. The facilitators)’ background, experience, and/or expertise 
enhanced the quality of the session. 

5. The facilitators)’ management style enhanced the 
quality of the session. 

6. The pace of the session was appropriate for the purposes of 
the professional development and the needs of adult learners. 

7. The session modeled effective assessment strategies. 

8 . 



Not 

at 

all 



2 

2 

2 



To a 
great 
extent 



1 2 3 4 5 



4 5 



1 2 3 4 5 



4 5 



4 5 



4 

4 

4 



Don’t 

know 



N/A 



7 

7 



B. Synthesis Rating 



1 


2 


3 


4 


5 


Implementation of the 
session not at all reflective 
of best practice for 
professional development. 








Implementation of the 
session extremely 
reflective of best practice 
for professional 
development 



C. Supporting Evidence for Synthesis Rating 
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HI. Mathematics/Science Content 

Complete this category if: a) increasing mathematics/science content knowledge was a key purpose of the session; b) 
mathematics/science content was a vehicle for accomplishing other professional development purposes; or c) inadequate 
coverage in this area acted as a barrier to accomplishing other stated purposes of the session. If none of these apply, 
check here □ and skip to category IV. 







Not 








To a 






A. 


Ratings of Key Indicators 


at 








great 


Don’t 








all 








extent 


know 


N/A 


1 . 


Mathematics/science content was appropriate for the purposes 


















of the professional development session and the backgrounds 


















of the participants. 


1 


2 


3 


4 


5 


6 


7 


2. 


Mathematics/science content was sound and appropriately 


















presented/explored. 


1 


2 


3 


4 


5 


6 


7 


3. 


Participants were intellectually engaged with important 


















ideas relevant to the focus of the session. 


1 


2 


3 


4 


5 


6 


7 


4. 


Facilitators) displayed an understanding of mathematics/science 


















concepts (e.g., in their dialogue with participants). 


1 


2 


3 


4 


5 


6 


7 


5. 


Mathematics/science was portrayed as a dynamic body of 


















knowledge continually enriched by conjecture, investigation, 


















analysis, and/or proof/justification. 


1 


2 


3 


4 


5 


6 


7 


6 . 


Depth and breadth of attention to mathematics/science content was 


















appropriate for the purposes of the session and participants’ needs. 


1 


2 


3 


4 


5 


6 


7 


7. 


Elements of mathematical/scientific abstraction (e.g., symbolic 


















representations, theory building) were included when it was 


















important to do so. 


1 


2 


3 


4 


5 


6 


7 


8. 


Appropriate connections were made to other areas of mathematics/ 


















science, to other disciplines, and/or to real-world contexts. 


1 


2 


3 


4 


5 


6 


7 


9. 


Extent of "sense-making" of mathematics/science content was 


















appropriate for the purposes of the session and the needs of adult 


















learners. 


1 


2 


3 


4 


5 


6 


7 


10. 




1 


2 


3 


4 


5 







B. Synthesis Rating 



1 


2 


3 


4 


5 1 


Mathematics/science 
content of session not at 
all reflective of current 
standards for 
mathematics/science 
education 








Mathematics/science 1 

content of session 
extremely reflective of 
current standards for 
mathematics/science 
education | 



C. Supporting Evidence for Synthesis Rating 
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IV. Exploring Pedagogy/Instructional Materials 



Complete this category if: a) exploring classroom practice/instructional materials was a key purpose of the session; orb) 
lack of/inadequate coverage in this area acted as a barrier to accomplishing other stated purposes of the session. If 
neither of these apply, check here □ and skip to category V. 





Not 






To a 






A. 


Ratings of Key Indicators 


at 






great 


Don’t 








all 






extent 


know 


N/A 


1 . 


Depth and breadth of attention to student thinking/leaming were 
















appropriate for the purposes of the session and participants’ needs. 


1 2 


3 


4 


5 


6 


7 


2. 


Depth and breadth of attention to classroom 
















strategies were appropriate for the purposes of the session and 
















participants’ needs. 


1 2 


3 


4 


5 


6 


7 


3. 


Depth and breadth of attention to instructional materials intended for 
















classroom use were appropriate for the purposes of the session and 
















participants’ needs. 


1 2 


3 


4 


5 


6 


7 


4. 


Facilitators) displayed an understanding of pedagogical 
















concepts (e.g., in their dialogue with participants). 


1 2 


3 


4 


5 


6 


7 


5. 


Participants were intellectually engaged with important 
















ideas relevant to classroom practice. 


1 2 


3 


4 


5 


6 


7 


6. 


Extent of "sense-making" about classroom practice was appropriate 
















for the purposes of the session and the needs of adult learners. 


1 2 


3 


4 


5 


6 


7 


7. 




1 2 


3 


4 


5 







B. Synthesis Rating 



1 


2 


3 


4 


5 


Pedagogical content of 
session not at all reflective . 
of current standards for 
mathematics/science 
education 








Pedagogical content of 
session extremely 
reflective of current 
standards for 
mathematics/science 
education 



C. Supporting Evidence for Synthesis Rating 
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V. Leadership Content 



Complete this category only if exploring strategies/issues/roles of teacher leaders, principals, or others in leadership 
positions was a key purpose of the session. If not, check here □ and skip to category VI. 



A. Ratings of Key Indicators 

1. Information on principles of effective staff development was 
sound and appropriately presented/explored. 



Not 

at 

all 



To a 

great Don’t 
extent know 



1 2 3 4 5 



6 



N/A 

7 



2. Information on strategies for mentoring/coaching peers was 
sound and appropriately presented/explored. 



2 3 4 5 



6 7 



3. Information on how to be a reform advocate at the school/district 
level was sound and appropriately presented/explored. 

4. Facilitator(s) displayed an understanding of leadership concepts 
(e.g., in their dialogue with participants). 

5. Participants were intellectually engaged with important 
ideas relevant to the focus of the session. 

6. Participants were given adequate and appropriate opportunity to 
consider how the content of the session applies to their particular 
leadership roles. 



1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 

1 2 3 4 5 



6 7 

6 7 

6 7 

6 7 



7. 



2 3 4 5 



B. Synthesis Rating 



1 


2 


3 


4 


5 


Leadership content not at 
all appropriate for 
preparing participants to 
be school/district leaders 
of mathematics/science . 
education 








Leadership content highly 
appropriate for preparing 
participants to be 
school/district leaders of 
mathematics/science 
education 



C. Supporting Evidence for Synthesis Rating 
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VI. Culture of the Professional Development Session 



Al. Ratings of Key Indicators 

1 . Active participation of all was encouraged and valued. 

2. There was a climate of respect for participants’ experiences, 
ideas, and contributions. 

3 . Interactions reflected collaborative working relationships 
among participants. 

4. Interactions reflected collaborative working relationships 
between facilitator(s) and participants. 

5. Participants were encouraged to generate ideas, questions, 
conjectures, and propositions. 

6. Participants demonstrated a willingness to share ideas and take 
intellectual risks. 

7. Intellectual rigor, constructive criticism, and the challenging 
of ideas were evident. 



Not 

at 

all 



To a 
great 
extent 



1 2 3 4 5 



1 2 3 4 5 



1 2 3 4 5 



1 2 3 4 5 



1 2 3 4 5 



123.45 



1 2 3 4 5 



Don’t 

know 



N/A 

7 



A2. Respect for Diversity 

Based on the culture of a professional development session, observers are generally able to make inferences about the 
extent to which there is an appreciation of diversity among participants (e.g., their gender, race/ethnicity, and/or 
cultural background). While direct evidence that reflects particular sensitivity or insensitivity toward diversity is not 
often observed, we would like you to document any examples you do see. If any examples were observed, please 
check here □ and describe below: 



B. Synthesis Rating 



1 


2 


3 


4 


5 


Culture of the session 
interfered with 
engagement of 
participants as members of 
a professional learning 
community 








Culture of the session 
facilitated engagement of 
participants as members of 
a professional learning 
community 



C. Supporting Evidence for Synthesis Rating 
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VII. Overall Ratings of the Session 



While the impact of a single professional development session may well be limited in scope, it is important to judge whether 
the session is likely to help move participants in the desired direction. For ratings in Sections A and B below, consider all 
available information (i.e., your previous ratings of design, implementation, content, and culture; related interviews; and your 
knowledge of the overall professional development program) as you assess the likely impact of this session. Feel free to 
elaborate on ratings with comments in the space provided. 

A. Likely Impact on Participants’ Capacity to Provide High Quality Mathematics/Science 
Education 

Consider the likely impact of this session on the participants’ capacity to provide high quality mathematics/science 
education. Select the response that best describes your overall assessment of the likely effect of this session in each of the 
following areas. 

□ Not applicable (The session did not focus on building capacity for classroom instruction.) 



Mixed or 





Negative 




Neutral 




Positive 


Don’t 






effect 




effect 




effect 


know 


N/A 


Participants’ ability to identify and understand important 
ideas of mathematics/science. 


o 


o 


o 


o 


o 


o 


o 


Participants’ understanding of mathematics/science as a dynamic 
body of knowledge generated and enriched by investigation. 


o 


o 


o 


o 


o 


o 


o 


Participants’ understanding of how students leam. 


o 


o 


o 


o 


o 


o 


o 


Participants’ ability to plan/provide high quality mathematics/ 
science classroom instruction. 


o 


o 


o 


o 


o 


o 


o 


Participants’ ability to use the designated instructional materials 
to develop students' conceptual understanding. 


o 


o 


o 


o 


o 


o 


o 


Participants’ self-confidence as mathematics/science instructors. 


o 


o 


o 


o 


o 


o 


o 


Professional networking among participants with regard to 
mathematics/science instruction. 


o 


o 


o 


o 


o 


o 


o 



Comments (optional): 
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B. 



1 . 

2 . 

3. 

4. 

5. 

6 . 

7. 

8 . 

9. 

10 . 



Likely Impact on Participants’ Leadership Capacity 

If the session included any teacher leaders, principals, or others in leadership positions, consider the likely impact of this 
session on their leadership capacity. Select the response that best describes your overall assessment of the likely effect of 
this session in each of the following areas. Please note that even if an element was not addressed explicitly, it might have 
a negative or positive effect on leadership development, depending on whether it was modeled well or poorly. 

□ Not applicable (The session did not include teacher leaders, principals, or others in leadership positions.) 









Mixed or 












Negative 




Neutral 




Positive 


Don’t 






effect 




effect 




effect 


know 


N/A 


Leaders’ knowledge and understanding of 
mathematics/science. 


o 


o 


o 


o 


o 


o 


o 


Leaders’ knowledge and understanding of effective classroom 
practice. 


o 


o 


o 


o 


o 


o 


o 


Leaders’ ability to convey to others a vision of effective 
mathematics/science classrooms. 


o 


o 


o 


o 


o 


o 


o 


Leaders’ understanding of teachers’ prior knowledge 
and areas where teachers have difficulty. 


o 


o 


o 


o 


o 


o 


o 


Leaders’ understanding of adult learners. 


o 


o 


o 


o 


o 


o 


o 


Leaders’ understanding of the reform process. 


o 


o 


o 


o 


o 


o 


o 


Leaders’ understanding of important strategies for reform 
of mathematics/science education. 


o 


o 


o 


o 


o 


o 


o 


Leaders’ ability to plan/implement exemplary 
professional development. 


o 


o 


o 


o 


o 


o 


o 


Leaders’ confidence in serving in leadership roles. 


o 


o 


o 


o 


o 


o 


o 


Professional networking among leaders with regard to 
leadership issues. 


o 


o 


o 


o 


o 


o 


o 



Comments (optional): 
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C. Capsule Description of the Quality of the Professional Development Session 

In this final rating of the session, consider all available information about the session, its context and purpose, and your own 
judgment of the relative importance of the ratings you have made. Select the capsule description that best characterizes the 
session you observed. Keep in mind that this rating is not intended to be an average of all the previous ratings, but should 
encapsulate your overall assessment of the quality and likely impact of the session. Please provide a brief rationale for your final 
capsule description of the session in the space provided. 

O Level 1: Ineffective Professional Development 

There is little or no evidence of participant thinking or engagement with important ideas of mathematics/science education. 
Session is unlikely to enhance the capacity of participants to provide high quality mathematics/science education or to be 
effective leaders of mathematics/science education in the district(s). Professional development appears to be either (select 
one below): 

O Passive “Learning” 

Session is pedantic and uninspiring. Participants are passive recipients of information; material is presented in a way 
that is inaccessible to or inappropriate for many of the participants. 

O Activity for Activity’s Sake 

Participants are involved in hands-on activities or other individual or group work, but it appears to be activity for 
activity’s sake. Session lacks a clear sense of purpose and/or a clear link to the conceptual development of participants. 

O Level 2: Elements of Effective Professional Development 

Session contains some elements of effective practice in professional development, but there are substantial problems in the 
design, content, and/or implementation given the purposes of the session. For example, the content is presented in a way 
that would reinforce misconceptions or the pace is clearly too rapid for meaningful participant engagement. Overall, the 
session is quite limited in its likelihood to enhance the capacity of most participants to provide high quality 
mathematics/science education or to be effective leaders of mathematics/science education in the district(s). 

O Level 3: Beginning Stages of Effective Professional Development (Select one below.) 

O Low 3 O Solid 3 O High 3 

Professional development is purposeful and at times effective, but there are some weaknesses in the design, content, or 
implementation of the session. For example, participants’ expertise is not well-utilized; or participants are not given 
sufficient opportunity to reflect on what they are learning. Overall, the session is somewhat limited in its likelihood to 
enhance the capacity of participants to provide high quality mathematics/science education or to be effective leaders of 
mathematics/science education in the district(s). 

O Level 4: Accomplished, Effective Professional Development 

Facilitation is skillful and participants are engaged in purposeful work (e.g., investigations, discussions, presentations, 
reading) designed to deepen their understanding of important mathematics/science concepts; enhance their pedagogical 
skills and knowledge; increase their ability to use the designated instructional materials; or to enhance their leadership 
skills. The facilitator(s) implement the professional development session well and participants’ contributions are valued, 
but adaptation of content or format in response to participants’ needs and interests may be somewhat limited. The session is 
quite likely to enhance the capacity of most participants to provide high quality mathematics/science education or to be 
effective leaders of mathematics/science education in the district(s). 

O Level 5: Exemplary Professional Development 

Facilitation is skillful, and participants are highly engaged in purposeful work (e.g., investigations, discussions, 
presentations, reading) designed to deepen their understanding of important mathematics/science concepts; enhance their 
pedagogical skills and knowledge; increase their ability to use the designated instructional materials; or to enhance their 
leadership skills. The session is artfully implemented, with flexibility and responsiveness to participant needs/interests. 

The session is highly likely to enhance the capacity of participants to provide high quality mathematics/science education or 
to be effective leaders of mathematics/science education in the district(s). 

Please provide your rationale for the capsule rating: 
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1999-2000 Local Systemic Change 
Teacher Interview 1 



1 . What grade(s) do you teach? 

2. This district is involved in an NSF-supported local systemic change initiative. 2 

To what extent have you participated in those activities (e.g., number of hours/days since 
becoming involved in the project)? 3 

PROBE for both summer and academic year activities. 

3. How do you feel about the professional development provided by the LSC? 

4. How has the LSC affected you and your teaching? 

PROBE for examples of changes. 

5. What specific characteristics of the LSC have been most helpful to you? 

6. What aspects have been least helpful? Why? 

7. What else do you need in order to continue improving your mathematics (science) 
instruction? 

8. Sometimes school and district policies and practices facilitate reform. At other times they 
get in the way. Are there any policies or practices in your school or district that you believe 
will help you in making the changes suggested by the LSC? 



9. Are there any policies or practices that you believe will limit your ability to make the 
changes suggested by the LSC? 



This protocol should be used for teacher interviews in all projects, except those in the Baseline Year or Final Year. 

2 You may want to use the local name for the LSC instead of, or in addition to, mentioning NSF, perhaps even giving examples of specific 
activities. 

3 Only treated teachers who have participated in 20 or more hours of professional development have been included in the random sample for 
teacher interviews. 

78 



Horizon Research, Inc. 



1999-2000 Core Evaluation Manual: Teacher Interview - Page 1 



August 1999 



For teachers who have participated in LSC leadership development: 

(If teacher has not participated in LSC leadership development, SKIP to Question 11.) 

10. To what extent have the professional development activities prepared you for your role as a 
teacher leader of mathematics (science) reform in your school or district? 

PROBE for specific examples of preparedness. 

1 1 . Do you have any other comments you would like to share? 



v/ 
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Instructions: Please use a #2 pencil to complete this questionnaire. Darken ovals completely, but do not stray into adjacent ovals. 
Be sure to erase completely any stray marks. 

A. Teacher Demographic Information 



1. Are you: 


2. Rac 


O Male 




O Female 


'0 



o 



i - Are you: (Darken one or more.) 

American Indian or Alaskan Native O 

Asian O 

Black or African-American O 



Hispanic or Latino 

Native Hawaiian or Other Pacific Islander 
White 



3. How many college mathematics courses have you 
completed? (Darken one oval.) 

O None 
O 1 semester 
O 2 semesters 
O 3 semesters 
O; 4 semesters 
O 5 or more semesters 



4. Did your college mathematics coursework include 
the equivalent of at least one semester of: 



(Darken one oval on each line.) 


Yes 


# No 


a. 


Number system concepts 




sp. 


b. 


Concepts in algebra 


O 


O 


c : .S 


Concepts in geometry ; 


O 


-P 



5. How many years have you taught prior to this school year? (Darken one oval.) 

0-2 3-5 6-10 11-15 16-20 21-25 26 or more 

o'".; : o v; . * P ° 






The National Science Foundation’s Local Systemic Change (LSC) through Teacher Enhancement 



.'U'&t*' 



Program's Core Evaluation 






■ ■ 



|ij:f 



■ ' 






You have been selected to participate in the nationwide evaluation of the federally-funded Local Systemic Change (LSC) # 

program. LSC is a National Science Foundation Teacher Enhancement program that is currently funding more than 50 local 
projects that offer science and mathematics professional development to teachers around the country. The cover letter 
accompanying this questionnaire identifies the LSC proj ect in your area, as well as the instructional materials. that are 
the focus of that LSC project. '-4 ; v •" 






Several times over the course of the LSC, each project will ad[minister questionnaires to a sample of teachers who are targeted to 
participate in the local project's professional development activities. Note that you may be asked to complete this questionnaire fy 
even if you have not yet participated in the project's professional development; your response is important, regardless of whether 
you have already participated. A small number of randomly-selected teachers in each project is asked to provide additional * 
information in interviews, sometimes in conjunction with a classroom visit. In order to continue receiving federal funding, each g 
LSC project must participate in this national evaluation. ; - 

Data collection procedures have been developed to ensure high-quality data and protect teacher confidentiality. Your responses ?> 
will be kept strictly confidential; they will be combined with the responses of the other teachers in your project and used only for 
the LSC evaluation. The name label and numbering on this questionnaire are used to help local projects deliver questionnaires 5 
to the proper teachers and follow up with teachers who have not responded; no information identifying individual teachers will 
be reported under any circumstances. After you complete the questionnaire, you should remove the name label and return the 
questionnaire as specified by your local LSC project. Additional information about privacy, as well as public burden, is : 'yy 
provided on page 7 of this questionnaire. C •4/ ; . ' c V' /"i 1 
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B. Teacher Opinions and Preparedness 

6. Please provide your opinion about each of the following statements. 
(Darken one oval on each line.) 



Strongly Disagree g; 

Disagree 

No Opinion - 

Agree 

Strongly Agree 



a. Students generally learn mathematics best in classes with students of similar abilities. 

b. I feel supported by colleagues to try out new ideas in teaching mathematics. 

c. Teachers in this school have a shared vision of effective mathematics instruction. 

d. Teachers in this school regularly share ideas and materials related to mathematics. 

e. Teachers in this school are well-supplied with materials for investigative mathematics 
instruction. 

f. I have time during the regular school week to work with my peers on mathematics curriculum 
and instruction. 



CD CD CD CD CD 
CD CDCD GD CD 

c D©si©<i) 

CD CD CD CD CD 
CD CD CD GD CD 
CD CD CD CD CD 



g- 

h. 



j- 

k. 



I have adequate access to calculators for teaching mathematics. CD CD CD CD CD 

I have adequate access to computers for teaching mathematics. CD CD CD CD Cl) 

I enjoy teaching mathematics. CD CD CD CD CD 

I am well-informed about the NCTM Standards for the grades I teach. CD CD CD CD CD 

The mathematics program in this school is strongly supported by local organizations, institutions ; > • 

and/or businesses. CD CD CD CD © 



7. In the left section, please rate each of the following in terms of its importance for effective mathematics instruction 
in the grades you teach. In the right section, please indicate how prepared you feel to do each one. 

(Darken one oval in each section on each line.) 







Importance 






Preparation 














Not 




Fairly 


Very 




Not 


Somewhat 


Fairly 


Very 


Adequately 


Somewhat 


Well 


Well 


Important 


Important 


Important 


Important 


Prepared 


Prepared 


Prepared 


Prepare) 


a: Provide concrete experience before 










m 








abstract concepts, 
b. Develop students’ conceptual 


• \ CD • " • 


CD 


CD (£ 


;■ 










understanding of mathematics. 


CD 


CD 


CD 


CD 


CD 


(D 


CD 


CD 


c . Take students’ prior understanding 
into account when planning 


















curriculum and instruction. >•'* ; * * 

d. Practice computational skills and 


CD 1 


iffi IScd 




• CD |:> |f 




' ; " CD & 




v ;V-< d! 


algorithms. 


CD 


CD 


CD 


CD 


CD 


(D 


CD 


CD 


fe. Make connections between .... . .. 








• *, 1 K -W' 










mathematics arid other disciplines. • ; 




CD-v-;-.- 


ll' :'P : 


CDv|| 


iil 








f. Have students work in cooperative 


















learning groups. 


CD 


CD 


CD 


CD 


CD 


CD 


CD 


CD 


g. Have students participate in f • 
















■ • .- 5 >J: 


£.?. appropriate hands-on activities, 
h. Engage students in inquiry-oriented 


% cd; :/ ;,: 


:||gy; CD 






rlL 




L . -CD 




activities. 


CD 


CD 


CD 


CD 


CD 


CD 


CD 


CD 


i. Use calculators. 


g,c D 


Jf® cd;;l^.; : 












CD 


j. Use computers. 

k. Engage students in applications of 


CD 


CD 


CD 


CD 


CD 


CD 


CD 


CD 


mathematics in a variety of contexts. 


|;CD itf 




SU CD'-;- 


cd 


CD 


\ . j 


* 


i . ® 


1. Use performance-based assessment. 


CD 


CD 


CD 


CD 


CD 


CD 


CD 


CD 


m. Use portfolios.:. ^ 

n. Use informal questioning to assess 




cd?:: ' 




CD ^ 




• : /cd /./;;•} 


s: cd •;=? 




student understanding. 


CD 


CD 


CD 


CD 


CD 


CD 


(D 


CD 
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